2016-05-17 21:52:43,630 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-17 21:52:43,665 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-05-17 21:52:43,701 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-17 21:52:44,110 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-17 21:52:44,416 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-17 21:52:44,597 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 21:52:44,598 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 21:52:44,683 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: 2D07CA2DCFDA41B094045080A6B0423A
2016-05-17 21:52:44,991 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.178
2016-05-17 21:52:45,037 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.225
2016-05-17 21:52:45,039 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:45,040 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:45,041 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 21:52:45,186 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-05-17 21:52:45,402 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-05-17 21:52:45,705 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-17 21:52:45,707 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-17 21:52:45,708 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:45,708 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-05-17 21:52:45,708 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-17 21:52:45,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-17 21:52:45,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-17 21:52:45,711 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-05-17 21:52:45,711 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 21:52:45,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-17 21:52:45,912 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:45,918 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 21:52:45,918 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 21:52:45,958 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001 doesn't exist and is created
2016-05-17 21:52:46,015 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 21:52:46,015 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:46,016 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001
2016-05-17 21:52:46,044 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463521965917_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:46,085 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 21:52:46,099 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 21:52:46,116 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-17 21:52:46,116 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-17 21:52:46,116 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:46,121 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 21:52:46,124 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 21:52:46,127 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001/localmode-log-dir
2016-05-17 21:52:46,127 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001/localmode-log-dir/history.txt.appattempt_1463521965917_0001_000000, maxErrors=10
2016-05-17 21:52:46,128 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 21:52:46,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463521965917_0001_000000, appSubmitTime=1463521966031, launchTime=1463521966043
2016-05-17 21:52:46,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 21:52:46,135 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 21:52:46,139 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 21:52:46,140 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 21:52:46,196 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:44664
2016-05-17 21:52:46,217 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-17 21:52:46,217 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-17 21:52:46,217 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:46,238 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 21:52:46,305 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 21:52:46,305 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463521965917_0001_000000, startTime=1463521966304
2016-05-17 21:52:46,305 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 21:52:46,318 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 21:52:46,333 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 21:52:46,334 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463521965917_0001, dagName=[E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite
2016-05-17 21:52:46,535 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite
2016-05-17 21:52:46,636 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463521965917_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001/localmode-log-dir/dag_1463521965917_0001_1.dot
2016-05-17 21:52:46,640 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001/localmode-log-dir/dag_1463521965917_0001_1-tez-dag.pb.txt
2016-05-17 21:52:46,770 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521965917_0001_1
2016-05-17 21:52:46,770 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:46,770 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite
2016-05-17 21:52:46 Running Dag: dag_1463521965917_0001_1
2016-05-17 21:52:46,799 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 21:52:46,799 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463521965917_0001_1, submitTime=1463521966535
2016-05-17 21:52:46,828 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:46,835 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 21:52:46,837 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 21:52:46,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463521965917_0001_1, initTime=1463521966800
2016-05-17 21:52:46,838 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521965917_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 21:52:46,840 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:DAG_STARTED]: dagID=dag_1463521965917_0001_1, startTime=1463521966839
2016-05-17 21:52:46,840 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 21:52:46,841 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 21:52:46,842 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521965917_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 21:52:46,843 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: C40FD4688D654D48A898AC1416C8216F : {F451EB1F4F5340B3AF9D1755B09FAAE9={InputName=F451EB1F4F5340B3AF9D1755B09FAAE9}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 21:52:46,844 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: F451EB1F4F5340B3AF9D1755B09FAAE9, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 21:52:46,844 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:46,849 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463521965917_0001, dagName=[E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite
2016-05-17 21:52:46,852 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:46,852 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:46,855 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]. Starting root input initializers: 1
2016-05-17 21:52:46,864 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: F451EB1F4F5340B3AF9D1755B09FAAE9 on vertex vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:46,874 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 21:52:46,876 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:46,891 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463521965917_0001, with dag id: dag_1463521965917_0001_1
2016-05-17 21:52:46,897 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 21:52:46,899 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input F451EB1F4F5340B3AF9D1755B09FAAE9 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 21:52:46,900 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 21:52:46,900 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 21:52:46,902 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:46,902 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:46,920 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:46,922 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:46,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E378C7E864974B62A359CC9989943CE9, vertexId=vertex_1463521965917_0001_1_01, initRequestedTime=1463521966874, initedTime=1463521966921, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:46,923 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:46,923 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:46,952 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-05-17 21:52:46,957 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: F451EB1F4F5340B3AF9D1755B09FAAE9 on vertex vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:46,957 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:46,958 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:46,958 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:46,958 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:46,959 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A115C8C296604C3080CD01778C1D3F68, vertexId=vertex_1463521965917_0001_1_02, initRequestedTime=1463521966923, initedTime=1463521966958, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:46,959 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:46,960 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:46,960 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {F451EB1F4F5340B3AF9D1755B09FAAE9=forAllWorkUnits=true, update=[1]}
2016-05-17 21:52:46,961 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] parallelism set to 1
2016-05-17 21:52:46,961 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:46,972 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521965917_0001_1_00, eventCount=1
2016-05-17 21:52:46,974 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:46,975 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C40FD4688D654D48A898AC1416C8216F, vertexId=vertex_1463521965917_0001_1_00, initRequestedTime=1463521966842, initedTime=1463521966974, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 21:52:46,975 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] with distanceFromRoot: 0
2016-05-17 21:52:46,976 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 21:52:46,976 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,005 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:47,006 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,007 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,010 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,011 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=90E34D7A1BB143D7A1FA8E4914C04C0A, vertexId=vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 21:52:47,014 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=90E34D7A1BB143D7A1FA8E4914C04C0A, vertexId=vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,034 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=90E34D7A1BB143D7A1FA8E4914C04C0A, vertexName=B2E1BD088A4240E3A6C686E7471424AE, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 21:52:47,046 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=90E34D7A1BB143D7A1FA8E4914C04C0A, vertexId=vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:47,049 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:47,049 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, vertexId=vertex_1463521965917_0001_1_03, initRequestedTime=1463521966976, initedTime=1463521967049, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:47,050 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:47,053 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,053 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:47,055 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:47,056 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521965917_0001_1_00, startRequestedTime=1463521967051, startedTime=1463521967051
2016-05-17 21:52:47,057 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:47,090 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:47,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_STARTED]: vertexName=C40FD4688D654D48A898AC1416C8216F, taskId=task_1463521965917_0001_1_00_000000, scheduledTime=1463521967089, launchTime=1463521967089
2016-05-17 21:52:47,092 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:47,092 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521965917_0001_1_00 for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] numStartedSources: 1 numSources: 1
2016-05-17 21:52:47,092 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] with distanceFromRoot: 1
2016-05-17 21:52:47,096 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:47,101 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C40FD4688D654D48A898AC1416C8216F in vertex: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:47,105 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E378C7E864974B62A359CC9989943CE9 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:47,106 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:47,106 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521965917_0001_1_01, startRequestedTime=1463521967096, startedTime=1463521967096
2016-05-17 21:52:47,107 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:47,110 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: C40FD4688D654D48A898AC1416C8216F, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F451EB1F4F5340B3AF9D1755B09FAAE9, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:47,114 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:47,114 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521965917_0001_1_01 for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] numStartedSources: 1 numSources: 1
2016-05-17 21:52:47,114 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] with distanceFromRoot: 2
2016-05-17 21:52:47,115 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:47,115 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E378C7E864974B62A359CC9989943CE9 in vertex: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:47,116 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A115C8C296604C3080CD01778C1D3F68 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:47,116 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:47,117 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521965917_0001_1_02, startRequestedTime=1463521967115, startedTime=1463521967115
2016-05-17 21:52:47,117 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:47,117 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521965917_0001_1_02 for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] numStartedSources: 1 numSources: 1
2016-05-17 21:52:47,117 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] with distanceFromRoot: 3
2016-05-17 21:52:47,118 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A115C8C296604C3080CD01778C1D3F68 in vertex: B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:47,118 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B2E1BD088A4240E3A6C686E7471424AE with 1 source tasks and 1 pending tasks
2016-05-17 21:52:47,118 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:47,121 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521965917_0001_1_03, startRequestedTime=1463521967118, startedTime=1463521967118
2016-05-17 21:52:47,122 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:47,179 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 21:52:47,182 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 21:52:47,182 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 21:52:47,185 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521965917_0001_00_000001
2016-05-17 21:52:47,185 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:47,194 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521965917_0001_00_000001
2016-05-17 21:52:47,194 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 21:52:47,198 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:47,199 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:47,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521965917_0001_00_000001, launchTime=1463521967198
2016-05-17 21:52:47,202 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000001
2016-05-17 21:52:47,207 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521965917_0001_1_00_000000_0] to container: [container_1463521965917_0001_00_000001]
2016-05-17 21:52:47,208 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:47,211 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521965917_0001_00_000001 given task: attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,211 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521965917_0001_1_00_000000_0] started. Is using containerId: [container_1463521965917_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 21:52:47,211 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47 Starting to run new task attempt: attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,212 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,212 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:47,212 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C40FD4688D654D48A898AC1416C8216F, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0, startTime=1463521967209, containerId=container_1463521965917_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521965917_0001_00_000001/teamcity, completedLogs=
2016-05-17 21:52:47,212 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:47,214 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:47,214 INFO counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters: GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-17 21:52:47,215 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:47,215 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:47,216 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:47,217 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:47,217 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:47,244 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:47,244 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: C40FD4688D654D48A898AC1416C8216F, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F451EB1F4F5340B3AF9D1755B09FAAE9, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:47,247 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:47,252 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,253 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:47,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,257 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521965917_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 21:52:47,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=F451EB1F4F5340B3AF9D1755B09FAAE9, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 21:52:47,261 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,261 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:47,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:47,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:47,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,266 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,269 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:47,284 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:47,284 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:47,285 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 21:52:47,286 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:47,286 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,287 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:47,288 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,313 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C40FD4688D654D48A898AC1416C8216F, ordinal: 0
2016-05-17 21:52:47,314 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C40FD4688D654D48A898AC1416C8216F, mem on start (mb), free: 215, total: 281, max: 672
2016-05-17 21:52:47,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:47,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:47,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:47,315 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:47,319 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 21:52:47,320 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:47,320 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:47,321 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:47,321 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 21:52:47,322 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, F451EB1F4F5340B3AF9D1755B09FAAE9, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 21:52:47,322 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E378C7E864974B62A359CC9989943CE9, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 21:52:47,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:47,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,324 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:47,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: F451EB1F4F5340B3AF9D1755B09FAAE9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:47,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:47,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:47,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:47,327 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,329 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 21:52:47,342 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@1614de07
2016-05-17 21:52:47,357 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@1614de07
2016-05-17 21:52:47,358 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 21:52:47,359 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 21:52:47,367 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:47,397 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: F451EB1F4F5340B3AF9D1755B09FAAE9
2016-05-17 21:52:47,398 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: 93580E0C2E874FFD90CC20D8847D280A
2016-05-17 21:52:47,398 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 93580E0C2E874FFD90CC20D8847D280A
2016-05-17 21:52:47,433 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@3ad397b4; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@198cc898; comparator=cascading.tuple.hadoop.util.TupleComparator@69c58042; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:47,437 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:47,439 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:47,440 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:47,554 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 21:52:47,554 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 21:52:47,555 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 21:52:47,555 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 21:52:47,555 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 21:52:47,558 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:47,558 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:47,560 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C40FD4688D654D48A898AC1416C8216F, all 1 inputs ready in: 00:00:00.001
2016-05-17 21:52:47,570 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,571 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C40FD4688D654D48A898AC1416C8216F, mem on close (mb), free: 103, total: 281, max: 672
2016-05-17 21:52:47,571 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:47,571 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:47,572 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-05-17 21:52:47,572 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-05-17 21:52:47,589 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:47 Completed running task attempt: attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,608 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:47,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521965917_0001_1_00, eventCount=1
2016-05-17 21:52:47,612 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:47,618 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1637, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463521967560, Process_Duration=10, Process_End_Time=1463521967570, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-17 21:52:47,619 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C40FD4688D654D48A898AC1416C8216F, taskAttemptId=attempt_1463521965917_0001_1_00_000000_0, startTime=1463521967209, finishTime=1463521967610, timeTaken=401, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1637, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967560, Process_Duration=10, Process_End_Time=1463521967570, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:47,621 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:47,622 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:47,624 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:47,629 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:47,624 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521965917_0001_1_00_000000_0
2016-05-17 21:52:47,632 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000001
2016-05-17 21:52:47,635 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:47,639 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_FINISHED]: vertexName=C40FD4688D654D48A898AC1416C8216F, taskId=task_1463521965917_0001_1_00_000000, startTime=1463521967209, finishTime=1463521967633, timeTaken=424, status=SUCCEEDED, successfulAttemptID=attempt_1463521965917_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1637, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967560, Process_Duration=10, Process_End_Time=1463521967570, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:47,641 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:47,642 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] : 1
2016-05-17 21:52:47,642 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:47,650 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:47,653 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_FINISHED]: vertexName=C40FD4688D654D48A898AC1416C8216F, vertexId=vertex_1463521965917_0001_1_00, initRequestedTime=1463521966842, initedTime=1463521966974, startRequestedTime=1463521967051, startedTime=1463521967051, finishTime=1463521967643, timeTaken=592, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1637, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967560, Process_Duration=10, Process_End_Time=1463521967570, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463521967209, firstTasksToStart=[ task_1463521965917_0001_1_00_000000 ], lastTaskFinishTime=1463521967610, lastTasksToFinish=[ task_1463521965917_0001_1_00_000000 ], minTaskDuration=401, maxTaskDuration=401, avgTaskDuration=401.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521965917_0001_1_00_000000 ], longestDurationTasks=[ task_1463521965917_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:47,653 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:47,655 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:47,658 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:47,659 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521965917_0001_00_000001, stoppedTime=1463521967658, exitStatus=0
2016-05-17 21:52:47,660 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:47,660 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] attempt: attempt_1463521965917_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:47,661 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:47,661 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:47,663 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:47,663 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521965917_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 21:52:47,664 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_STARTED]: vertexName=E378C7E864974B62A359CC9989943CE9, taskId=task_1463521965917_0001_1_01_000000, scheduledTime=1463521967663, launchTime=1463521967663
2016-05-17 21:52:47,664 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521965917_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 21:52:47,664 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:47,664 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521965917_0001_00_000001, Exiting
2016-05-17 21:52:47,664 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:47,665 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521965917_0001_00_000001
2016-05-17 21:52:47,665 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521965917_0001_00_000001
2016-05-17 21:52:47,666 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:47,666 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:47,667 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:47,668 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E378C7E864974B62A359CC9989943CE9, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C40FD4688D654D48A898AC1416C8216F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:47,668 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521965917_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:47,668 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:47,668 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521965917_0001_00_000001 completed successfully
2016-05-17 21:52:47,670 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:47,673 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521965917_0001_00_000002
2016-05-17 21:52:47,674 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:47,675 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:47,678 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521965917_0001_00_000002
2016-05-17 21:52:47,678 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:47,680 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:47,681 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:47,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521965917_0001_00_000002, launchTime=1463521967681
2016-05-17 21:52:47,682 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000002
2016-05-17 21:52:47,683 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521965917_0001_1_01_000000_0] to container: [container_1463521965917_0001_00_000002]
2016-05-17 21:52:47,683 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:47,684 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521965917_0001_1_01_000000_0] started. Is using containerId: [container_1463521965917_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 21:52:47,684 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:47,684 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E378C7E864974B62A359CC9989943CE9, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0, startTime=1463521967684, containerId=container_1463521965917_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521965917_0001_00_000002/teamcity, completedLogs=
2016-05-17 21:52:47,684 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:47,685 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:47,686 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521965917_0001_00_000002 given task: attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,687 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47 Starting to run new task attempt: attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,689 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,690 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:47,690 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:47,691 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:47,691 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:47,691 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:47,692 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:47,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E378C7E864974B62A359CC9989943CE9, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C40FD4688D654D48A898AC1416C8216F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:47,693 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:47,694 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:47,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,695 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521965917_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 21:52:47,696 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,696 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:47,697 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:47,699 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:47,698 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C40FD4688D654D48A898AC1416C8216F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:47,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:47,703 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:47,703 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:47,704 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:47,705 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:47,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,721 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:47,722 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:47,722 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,723 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,727 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E378C7E864974B62A359CC9989943CE9, ordinal: 1
2016-05-17 21:52:47,727 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E378C7E864974B62A359CC9989943CE9, mem on start (mb), free: 90, total: 281, max: 672
2016-05-17 21:52:47,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:47,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:47,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:47,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:47,728 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:47,729 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:47,729 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:47,730 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:47,730 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:47,730 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A115C8C296604C3080CD01778C1D3F68, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:47,731 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C40FD4688D654D48A898AC1416C8216F, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:47,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:47,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C40FD4688D654D48A898AC1416C8216F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:47,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:47,732 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:47,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,742 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:47,743 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:47,747 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C40FD4688D654D48A898AC1416C8216F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:47,749 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:47,749 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:47,755 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:47,759 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,759 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:47,760 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,761 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521965917_0001_1_00_000000_0_10003, runDuration: 324314, ]
2016-05-17 21:52:47,766 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-05-17 21:52:47,770 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521965917_0001_1_00_000000_0_10003] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1463521967769, TimeTaken=2, Rate=0.52 MB/s
2016-05-17 21:52:47,770 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,771 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:47,771 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [C40FD4688D654D48A898AC1416C8216F] #1 in 5ms
2016-05-17 21:52:47,773 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C40FD4688D654D48A898AC1416C8216F] #1, status:false, isInterrupted:false
2016-05-17 21:52:47,775 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:47,776 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 93580E0C2E874FFD90CC20D8847D280A
2016-05-17 21:52:47,776 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: C724BBC0A33144F3A1D38B13A8D73374
2016-05-17 21:52:47,777 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C724BBC0A33144F3A1D38B13A8D73374
2016-05-17 21:52:47,780 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-05-17 21:52:47,781 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:47,783 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@42ff1452; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@788f35da; comparator=cascading.tuple.hadoop.util.TupleComparator@27107ade; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:47,784 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:47,784 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:47,785 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:47,787 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:47,852 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:47,853 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:47,853 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:47,853 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:47,854 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:47,855 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 93580E0C2E874FFD90CC20D8847D280A, for 1 inputs
2016-05-17 21:52:47,858 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-17 21:52:47,858 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,858 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:47,862 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:47,862 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:47,867 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E378C7E864974B62A359CC9989943CE9, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:47,871 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@b8519f4; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@4f6ae814
2016-05-17 21:52:47,912 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,912 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E378C7E864974B62A359CC9989943CE9, mem on close (mb), free: 82, total: 345, max: 672
2016-05-17 21:52:47,912 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C40FD4688D654D48A898AC1416C8216F
2016-05-17 21:52:47,913 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:47,913 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:47,914 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:47,914 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:47,914 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:47,915 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:47,915 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:47,915 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-05-17 21:52:47,916 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-05-17 21:52:47,951 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:47 Completed running task attempt: attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,956 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:47,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521965917_0001_1_01, eventCount=1
2016-05-17 21:52:47,958 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:47,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=28, MERGE_PHASE_TIME=115, FIRST_EVENT_RECEIVED=21, LAST_EVENT_RECEIVED=21][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521967867, Process_Duration=45, Process_End_Time=1463521967912, Read_Duration=5, Tuples_Read=35, Tuples_Written=174, Write_Duration=12]]
2016-05-17 21:52:47,961 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:47,961 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:47,961 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000002
2016-05-17 21:52:47,963 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E378C7E864974B62A359CC9989943CE9, taskAttemptId=attempt_1463521965917_0001_1_01_000000_0, startTime=1463521967684, finishTime=1463521967957, timeTaken=273, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=28, MERGE_PHASE_TIME=115, FIRST_EVENT_RECEIVED=21, LAST_EVENT_RECEIVED=21, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967867, Process_Duration=45, Process_End_Time=1463521967912, Read_Duration=5, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-05-17 21:52:47,963 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:47,964 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521965917_0001_1_01_000000_0
2016-05-17 21:52:47,964 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:47,965 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:47,966 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_FINISHED]: vertexName=E378C7E864974B62A359CC9989943CE9, taskId=task_1463521965917_0001_1_01_000000, startTime=1463521967684, finishTime=1463521967964, timeTaken=280, status=SUCCEEDED, successfulAttemptID=attempt_1463521965917_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=28, MERGE_PHASE_TIME=115, FIRST_EVENT_RECEIVED=21, LAST_EVENT_RECEIVED=21, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967867, Process_Duration=45, Process_End_Time=1463521967912, Read_Duration=5, Tuples_Read=35, Tuples_Written=174, Write_Duration=12
2016-05-17 21:52:47,967 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:47,967 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] : 1
2016-05-17 21:52:47,968 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:47,969 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:47,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_FINISHED]: vertexName=E378C7E864974B62A359CC9989943CE9, vertexId=vertex_1463521965917_0001_1_01, initRequestedTime=1463521966874, initedTime=1463521966921, startRequestedTime=1463521967096, startedTime=1463521967096, finishTime=1463521967968, timeTaken=872, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=28, MERGE_PHASE_TIME=115, FIRST_EVENT_RECEIVED=21, LAST_EVENT_RECEIVED=21, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521967867, Process_Duration=45, Process_End_Time=1463521967912, Read_Duration=5, Tuples_Read=35, Tuples_Written=174, Write_Duration=12, vertexStats=firstTaskStartTime=1463521967684, firstTasksToStart=[ task_1463521965917_0001_1_01_000000 ], lastTaskFinishTime=1463521967957, lastTasksToFinish=[ task_1463521965917_0001_1_01_000000 ], minTaskDuration=273, maxTaskDuration=273, avgTaskDuration=273.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521965917_0001_1_01_000000 ], longestDurationTasks=[ task_1463521965917_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:47,974 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:47,975 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:47,977 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521965917_0001_00_000002, stoppedTime=1463521967977, exitStatus=0
2016-05-17 21:52:47,977 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:47,978 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] attempt: attempt_1463521965917_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:47,979 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:47,977 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:47,980 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:47,981 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:47,981 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521965917_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-17 21:52:47,981 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_STARTED]: vertexName=A115C8C296604C3080CD01778C1D3F68, taskId=task_1463521965917_0001_1_02_000000, scheduledTime=1463521967981, launchTime=1463521967981
2016-05-17 21:52:47,982 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:47,982 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:47,982 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:47,982 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A115C8C296604C3080CD01778C1D3F68, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B2E1BD088A4240E3A6C686E7471424AE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:47,983 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:47,981 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:47,983 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:47,984 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521965917_0001_00_000003
2016-05-17 21:52:47,984 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:47,986 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521965917_0001_00_000003
2016-05-17 21:52:47,987 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-05-17 21:52:47,987 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:47,988 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:47,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521965917_0001_00_000003, launchTime=1463521967987
2016-05-17 21:52:47,989 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000003
2016-05-17 21:52:47,990 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521965917_0001_1_02_000000_0] to container: [container_1463521965917_0001_00_000003]
2016-05-17 21:52:47,990 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:47,990 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521965917_0001_00_000003 given task: attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:47,990 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:47,991 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521965917_0001_1_02_000000_0] started. Is using containerId: [container_1463521965917_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-17 21:52:47,992 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:47,992 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A115C8C296604C3080CD01778C1D3F68, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0, startTime=1463521967990, containerId=container_1463521965917_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521965917_0001_00_000003/teamcity, completedLogs=
2016-05-17 21:52:47,993 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:47 Starting to run new task attempt: attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:47,993 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:47,993 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:47,993 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:47,994 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:47,993 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:47,995 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:47,994 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521965917_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 21:52:47,996 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000002: 32 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:47,996 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521965917_0001_00_000002, Exiting
2016-05-17 21:52:47,996 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521965917_0001_00_000002
2016-05-17 21:52:47,996 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521965917_0001_00_000002
2016-05-17 21:52:47,998 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:47,998 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521965917_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:47,998 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521965917_0001_00_000002 completed successfully
2016-05-17 21:52:47,998 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:47,999 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:47,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A115C8C296604C3080CD01778C1D3F68, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B2E1BD088A4240E3A6C686E7471424AE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:48,000 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:48,000 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:48,002 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:48,002 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,005 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521965917_0001_1_02_000000_0 sent events: (0-1)
2016-05-17 21:52:48,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E378C7E864974B62A359CC9989943CE9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:48,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B2E1BD088A4240E3A6C686E7471424AE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:48,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:48,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,007 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:48,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:48,010 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:48,012 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:48,012 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:48,012 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:48,015 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:48,016 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:48,017 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,017 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,028 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A115C8C296604C3080CD01778C1D3F68, ordinal: 2
2016-05-17 21:52:48,028 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A115C8C296604C3080CD01778C1D3F68, mem on start (mb), free: 75, total: 345, max: 672
2016-05-17 21:52:48,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:48,029 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:48,029 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:48,029 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:48,029 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:48,029 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:48,029 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:48,030 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:48,030 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B2E1BD088A4240E3A6C686E7471424AE, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:48,030 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E378C7E864974B62A359CC9989943CE9, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:48,030 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:48,031 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E378C7E864974B62A359CC9989943CE9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:48,031 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,031 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:48,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:48,034 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:48,034 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:48,035 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E378C7E864974B62A359CC9989943CE9 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:48,036 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:48,037 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:48,038 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:48,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:48,040 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:48,040 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521965917_0001_1_01_000000_0_10005, runDuration: 254364, ]
2016-05-17 21:52:48,042 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521965917_0001_1_01_000000_0_10005] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1463521968042, TimeTaken=1, Rate=7.17 MB/s
2016-05-17 21:52:48,042 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,042 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-17 21:52:48,043 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [E378C7E864974B62A359CC9989943CE9] #2 in 2ms
2016-05-17 21:52:48,043 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E378C7E864974B62A359CC9989943CE9] #2, status:false, isInterrupted:false
2016-05-17 21:52:48,044 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-17 21:52:48,044 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:48,045 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-05-17 21:52:48,046 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:48,046 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: C724BBC0A33144F3A1D38B13A8D73374
2016-05-17 21:52:48,046 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: 8F1204ACA49241A4B556D5F4BE1FA65E
2016-05-17 21:52:48,046 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:48,046 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 8F1204ACA49241A4B556D5F4BE1FA65E
2016-05-17 21:52:48,049 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-17 21:52:48,049 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,050 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:48,052 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@428d0463; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@381b21b7; comparator=cascading.tuple.hadoop.util.TupleComparator@1d897eb0; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:48,053 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:48,053 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:48,053 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:48,126 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:48,127 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:48,127 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:48,127 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:48,127 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:48,128 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C724BBC0A33144F3A1D38B13A8D73374, for 1 inputs
2016-05-17 21:52:48,128 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A115C8C296604C3080CD01778C1D3F68, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:48,130 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@2dd72f1f; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@a56b3bc
2016-05-17 21:52:48,158 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:48,158 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A115C8C296604C3080CD01778C1D3F68, mem on close (mb), free: 71, total: 408, max: 672
2016-05-17 21:52:48,158 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E378C7E864974B62A359CC9989943CE9
2016-05-17 21:52:48,159 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:48,160 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:48,160 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:48,160 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:48,161 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:48,161 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:48,161 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:48,161 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-05-17 21:52:48,162 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-05-17 21:52:48,171 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:48 Completed running task attempt: attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:48,178 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:48,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521965917_0001_1_02, eventCount=1
2016-05-17 21:52:48,182 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521968128, Process_Duration=30, Process_End_Time=1463521968158, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=4]]
2016-05-17 21:52:48,182 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:48,182 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:48,185 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A115C8C296604C3080CD01778C1D3F68, taskAttemptId=attempt_1463521965917_0001_1_02_000000_0, startTime=1463521967990, finishTime=1463521968182, timeTaken=192, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968128, Process_Duration=30, Process_End_Time=1463521968158, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-05-17 21:52:48,184 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:48,185 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:48,188 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:48,185 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000003
2016-05-17 21:52:48,188 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521965917_0001_1_02_000000_0
2016-05-17 21:52:48,189 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:48,191 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_FINISHED]: vertexName=A115C8C296604C3080CD01778C1D3F68, taskId=task_1463521965917_0001_1_02_000000, startTime=1463521967990, finishTime=1463521968188, timeTaken=198, status=SUCCEEDED, successfulAttemptID=attempt_1463521965917_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968128, Process_Duration=30, Process_End_Time=1463521968158, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-05-17 21:52:48,191 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:48,193 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] : 1
2016-05-17 21:52:48,194 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:48,195 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:48,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_FINISHED]: vertexName=A115C8C296604C3080CD01778C1D3F68, vertexId=vertex_1463521965917_0001_1_02, initRequestedTime=1463521966923, initedTime=1463521966958, startRequestedTime=1463521967115, startedTime=1463521967115, finishTime=1463521968194, timeTaken=1079, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968128, Process_Duration=30, Process_End_Time=1463521968158, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=4, vertexStats=firstTaskStartTime=1463521967990, firstTasksToStart=[ task_1463521965917_0001_1_02_000000 ], lastTaskFinishTime=1463521968182, lastTasksToFinish=[ task_1463521965917_0001_1_02_000000 ], minTaskDuration=192, maxTaskDuration=192, avgTaskDuration=192.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521965917_0001_1_02_000000 ], longestDurationTasks=[ task_1463521965917_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:48,197 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:48,197 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:48,198 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:48,198 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521965917_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-17 21:52:48,199 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:48,198 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521965917_0001_00_000003, stoppedTime=1463521968198, exitStatus=0
2016-05-17 21:52:48,199 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521965917_0001_00_000003, Exiting
2016-05-17 21:52:48,199 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:48,199 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] attempt: attempt_1463521965917_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:48,200 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B2E1BD088A4240E3A6C686E7471424AE
2016-05-17 21:52:48,199 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521965917_0001_00_000003
2016-05-17 21:52:48,200 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521965917_0001_00_000003
2016-05-17 21:52:48,200 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:48,200 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521965917_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:48,201 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:48,201 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521965917_0001_00_000003
2016-05-17 21:52:48,201 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521965917_0001_00_000003 completed successfully
2016-05-17 21:52:48,203 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:48,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_STARTED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, taskId=task_1463521965917_0001_1_03_000000, scheduledTime=1463521968203, launchTime=1463521968203
2016-05-17 21:52:48,204 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:48,204 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:48,204 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:48,205 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:48,205 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: B2E1BD088A4240E3A6C686E7471424AE, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=90E34D7A1BB143D7A1FA8E4914C04C0A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:48,205 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:48,206 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:48,207 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521965917_0001_00_000004
2016-05-17 21:52:48,207 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:48,209 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521965917_0001_00_000004
2016-05-17 21:52:48,210 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:48,213 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:48,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521965917_0001_00_000004, launchTime=1463521968213
2016-05-17 21:52:48,213 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:48,214 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000004
2016-05-17 21:52:48,216 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521965917_0001_1_03_000000_0] to container: [container_1463521965917_0001_00_000004]
2016-05-17 21:52:48,216 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:48,216 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521965917_0001_00_000004 given task: attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,217 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521965917_0001_1_03_000000_0] started. Is using containerId: [container_1463521965917_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-17 21:52:48,217 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,217 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:48 Starting to run new task attempt: attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,218 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,218 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:48,218 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:48,218 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:48,218 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:48,218 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:48,218 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:48,219 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C] euclidean/(1/1) ...osite/euclidean/composite, VertexName: B2E1BD088A4240E3A6C686E7471424AE, VertexParallelism: 1, TaskAttemptID:attempt_1463521965917_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=90E34D7A1BB143D7A1FA8E4914C04C0A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:48,219 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:48,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0, startTime=1463521968216, containerId=container_1463521965917_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521965917_0001_00_000004/teamcity, completedLogs=
2016-05-17 21:52:48,220 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:48,220 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:48,220 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,220 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:48,221 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,222 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A115C8C296604C3080CD01778C1D3F68, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:48,225 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521965917_0001_1_03_000000_0 sent events: (0-1)
2016-05-17 21:52:48,224 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=90E34D7A1BB143D7A1FA8E4914C04C0A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 21:52:48,222 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:48,228 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,231 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:48,231 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:48,232 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,231 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:48,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:48,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 90E34D7A1BB143D7A1FA8E4914C04C0A
2016-05-17 21:52:48,234 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 21:52:48,235 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 90E34D7A1BB143D7A1FA8E4914C04C0A
2016-05-17 21:52:48,255 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 21:52:48,256 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 90E34D7A1BB143D7A1FA8E4914C04C0A
2016-05-17 21:52:48,257 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B2E1BD088A4240E3A6C686E7471424AE, ordinal: 3
2016-05-17 21:52:48,257 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2E1BD088A4240E3A6C686E7471424AE, mem on start (mb), free: 64, total: 408, max: 672
2016-05-17 21:52:48,259 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:48,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:48,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:48,261 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:48,261 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:48,262 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:48,263 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 21:52:48,263 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:48,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:48,265 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 21:52:48,265 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A115C8C296604C3080CD01778C1D3F68, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 21:52:48,266 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 90E34D7A1BB143D7A1FA8E4914C04C0A, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 21:52:48,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:48,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,268 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:48,269 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:48,269 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A115C8C296604C3080CD01778C1D3F68 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:48,269 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 21:52:48,269 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:48,270 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:48,270 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A115C8C296604C3080CD01778C1D3F68 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:48,271 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,273 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:48,275 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:48,275 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:48,276 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,276 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521965917_0001_1_02_000000_0_10009, runDuration: 167070, ]
2016-05-17 21:52:48,279 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521965917_0001_1_02_000000_0_10009] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1463521968279, TimeTaken=2, Rate=2.07 MB/s
2016-05-17 21:52:48,279 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,280 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:48,281 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [A115C8C296604C3080CD01778C1D3F68] #3 in 4ms
2016-05-17 21:52:48,282 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-05-17 21:52:48,283 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A115C8C296604C3080CD01778C1D3F68] #3, status:false, isInterrupted:false
2016-05-17 21:52:48,284 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:48,285 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: 8F1204ACA49241A4B556D5F4BE1FA65E
2016-05-17 21:52:48,286 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-05-17 21:52:48,286 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:48,286 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:48,286 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: 90E34D7A1BB143D7A1FA8E4914C04C0A
2016-05-17 21:52:48,287 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-05-17 21:52:48,288 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,290 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:48,290 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-05-17 21:52:48,291 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 8F1204ACA49241A4B556D5F4BE1FA65E, for 1 inputs
2016-05-17 21:52:48,291 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B2E1BD088A4240E3A6C686E7471424AE, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:48,293 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@4d957e26; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@118d1895
2016-05-17 21:52:48,305 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 21:52:48,305 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 21:52:48,306 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,307 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463521965917_0001_1_03_000000_0 given a go for committing the task output.
2016-05-17 21:52:48,308 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,308 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2E1BD088A4240E3A6C686E7471424AE, mem on close (mb), free: 60, total: 408, max: 672
2016-05-17 21:52:48,309 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A115C8C296604C3080CD01778C1D3F68
2016-05-17 21:52:48,310 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:48,310 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:48,310 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:48,311 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:48,311 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:48 Completed running task attempt: attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,314 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:48,315 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:48,318 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, taskAttemptId=attempt_1463521965917_0001_1_03_000000_0, startTime=1463521968216, finishTime=1463521968315, timeTaken=99, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968292, Process_Duration=16, Process_End_Time=1463521968308, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:48,318 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521965917_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:48,318 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521965917_0001_1_03_000000_0
2016-05-17 21:52:48,318 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:48,319 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:48,320 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:TASK_FINISHED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, taskId=task_1463521965917_0001_1_03_000000, startTime=1463521968216, finishTime=1463521968318, timeTaken=102, status=SUCCEEDED, successfulAttemptID=attempt_1463521965917_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968292, Process_Duration=16, Process_End_Time=1463521968308, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:48,320 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521965917_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:48,321 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] : 1
2016-05-17 21:52:48,321 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:48,322 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:48,324 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:VERTEX_FINISHED]: vertexName=B2E1BD088A4240E3A6C686E7471424AE, vertexId=vertex_1463521965917_0001_1_03, initRequestedTime=1463521966976, initedTime=1463521967049, startRequestedTime=1463521967118, startedTime=1463521967118, finishTime=1463521968321, timeTaken=1203, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521968292, Process_Duration=16, Process_End_Time=1463521968308, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463521968216, firstTasksToStart=[ task_1463521965917_0001_1_03_000000 ], lastTaskFinishTime=1463521968315, lastTasksToFinish=[ task_1463521965917_0001_1_03_000000 ], minTaskDuration=99, maxTaskDuration=99, avgTaskDuration=99.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521965917_0001_1_03_000000 ], longestDurationTasks=[ task_1463521965917_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:48,324 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:48,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=900, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521968292, Process_Duration=16, Process_End_Time=1463521968308, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-17 21:52:48,325 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:48,325 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:48,326 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:48,326 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521965917_0001_00_000004
2016-05-17 21:52:48,326 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:48,326 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521965917_0001_00_000004, stoppedTime=1463521968326, exitStatus=0
2016-05-17 21:52:48,327 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:48,327 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:48,328 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463521965917_0001_1
2016-05-17 21:52:48,327 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521965917_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-17 21:52:48,328 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521965917_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:48,329 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521965917_0001_00_000004, Exiting
2016-05-17 21:52:48,329 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521965917_0001_00_000004
2016-05-17 21:52:48,329 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521965917_0001_00_000004
2016-05-17 21:52:48,330 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521965917_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:48,330 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521965917_0001_00_000004 completed successfully
2016-05-17 21:52:48,332 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463521965917_0001_1
2016-05-17 21:52:48,333 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521965917_0001_1_01 [E378C7E864974B62A359CC9989943CE9]
2016-05-17 21:52:48,333 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521965917_0001_1_00 [C40FD4688D654D48A898AC1416C8216F]
2016-05-17 21:52:48,333 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 90E34D7A1BB143D7A1FA8E4914C04C0A for vertex: vertex_1463521965917_0001_1_03 [B2E1BD088A4240E3A6C686E7471424AE]
2016-05-17 21:52:48,349 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521965917_0001_1_02 [A115C8C296604C3080CD01778C1D3F68]
2016-05-17 21:52:48,358 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463521965917_0001_1, queueSize=0
2016-05-17 21:52:48,359 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 21:52:48,361 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521965917_0001_1][Event:DAG_FINISHED]: dagId=dag_1463521965917_0001_1, startTime=1463521966839, finishTime=1463521968350, timeTaken=1511, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14464, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1513619456, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=48, MERGE_PHASE_TIME=148, FIRST_EVENT_RECEIVED=35, LAST_EVENT_RECEIVED=35, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854087871847, Process_Duration=101, Process_End_Time=5854087871948, Read_Duration=22, Tuples_Read=303, Tuples_Written=317, Write_Duration=19, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-17 21:52:48,363 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463521965917_0001_1 finished with state: SUCCEEDED
2016-05-17 21:52:48,363 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521965917_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 21:52:48,363 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521965917_0001_00_000004
2016-05-17 21:52:48,363 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521965917_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:48 Completed Dag: dag_1463521965917_0001_1
2016-05-17 21:52:48,365 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463521965917_0001_1, dagState=SUCCEEDED
2016-05-17 21:52:48,365 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521965917_0001_1_post
2016-05-17 21:52:48,365 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:48,366 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 21:52:48,376 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463521965917_0001
2016-05-17 21:52:48,379 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 21:52:48,380 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 21:52:48,380 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 21:52:48,380 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 21:52:48,380 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 21:52:48,381 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 21:52:48,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 21:52:48,384 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 21:52:48,384 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 21:52:48,384 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 21:52:48,384 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 21:52:48,385 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 21:52:48,385 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 21:52:48,391 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/E037DB4863F342509568D6DB9BC4E1D5/EE7CD50DC6594305976E7F388E57D90C/.tez/application_1463521965917_0001
2016-05-17 21:52:48,392 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite/_temporary
2016-05-17 21:52:48,393 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:02.684
2016-05-17 21:52:48,544 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-05-17 21:52:48,587 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 21:52:48,587 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 21:52:48,696 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.066
2016-05-17 21:52:48,726 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.096
2016-05-17 21:52:48,727 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:48,727 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:48,727 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 21:52:48,752 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-05-17 21:52:48,786 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-05-17 21:52:48,989 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] starting
2016-05-17 21:52:48,990 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:48,990 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-05-17 21:52:48,990 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] parallel execution of steps is enabled: false
2016-05-17 21:52:48,991 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executing total steps: 1
2016-05-17 21:52:48,991 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] allocating management threads: 1
2016-05-17 21:52:48,992 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-05-17 21:52:48,993 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 21:52:49,009 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tez session mode enabled: true
2016-05-17 21:52:49,014 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:49,015 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 21:52:49,015 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 21:52:49,026 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001 doesn't exist and is created
2016-05-17 21:52:49,046 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 21:52:49,046 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:49,047 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001
2016-05-17 21:52:49,053 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463521969015_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:49,068 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 21:52:49,069 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 21:52:49,071 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 21:52:49,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 21:52:49,072 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001/localmode-log-dir
2016-05-17 21:52:49,072 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001/localmode-log-dir/history.txt.appattempt_1463521969015_0001_000000, maxErrors=10
2016-05-17 21:52:49,072 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 21:52:49,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463521969015_0001_000000, appSubmitTime=1463521969052, launchTime=1463521969053
2016-05-17 21:52:49,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 21:52:49,082 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 21:52:49,084 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 21:52:49,085 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:40635
2016-05-17 21:52:49,085 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 21:52:49,089 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 21:52:49,092 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 21:52:49,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463521969015_0001_000000, startTime=1463521969092
2016-05-17 21:52:49,092 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 21:52:49,147 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 21:52:49,147 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 21:52:49,147 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463521969015_0001, dagName=[EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite
2016-05-17 21:52:49,169 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite
2016-05-17 21:52:49,177 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463521969015_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001/localmode-log-dir/dag_1463521969015_0001_1.dot
2016-05-17 21:52:49,182 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001/localmode-log-dir/dag_1463521969015_0001_1-tez-dag.pb.txt
2016-05-17 21:52:49,227 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521969015_0001_1
2016-05-17 21:52:49,227 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,227 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite
2016-05-17 21:52:49 Running Dag: dag_1463521969015_0001_1
2016-05-17 21:52:49,250 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463521969015_0001_1, submitTime=1463521969169
2016-05-17 21:52:49,250 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 21:52:49,251 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,251 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 21:52:49,252 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 21:52:49,254 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463521969015_0001_1, initTime=1463521969250
2016-05-17 21:52:49,254 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521969015_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 21:52:49,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:DAG_STARTED]: dagID=dag_1463521969015_0001_1, startTime=1463521969254
2016-05-17 21:52:49,255 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 21:52:49,255 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521969015_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 21:52:49,255 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: B6FC4A96F0D14AD8A08DCB78D2443B04 : {1DFCDDDFA1784698BFBDD8A3F6120887={InputName=1DFCDDDFA1784698BFBDD8A3F6120887}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 21:52:49,255 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 1DFCDDDFA1784698BFBDD8A3F6120887, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 21:52:49,255 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,256 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,256 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,255 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 21:52:49,256 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]. Starting root input initializers: 1
2016-05-17 21:52:49,257 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 21:52:49,257 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,260 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 1DFCDDDFA1784698BFBDD8A3F6120887 on vertex vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,263 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463521969015_0001, dagName=[EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite
2016-05-17 21:52:49,264 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:49,264 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,265 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 21:52:49,265 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,266 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:49,266 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, vertexId=vertex_1463521969015_0001_1_01, initRequestedTime=1463521969257, initedTime=1463521969265, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:49,266 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:49,266 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,267 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 1DFCDDDFA1784698BFBDD8A3F6120887 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 21:52:49,303 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 21:52:49,305 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 21:52:49,309 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-05-17 21:52:49,309 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 1DFCDDDFA1784698BFBDD8A3F6120887 on vertex vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,311 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:49,311 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,312 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,312 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:49,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, vertexId=vertex_1463521969015_0001_1_02, initRequestedTime=1463521969266, initedTime=1463521969312, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:49,313 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:49,313 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:49,313 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {1DFCDDDFA1784698BFBDD8A3F6120887=forAllWorkUnits=true, update=[1]}
2016-05-17 21:52:49,314 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] parallelism set to 1
2016-05-17 21:52:49,314 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:49,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521969015_0001_1_00, eventCount=1
2016-05-17 21:52:49,316 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:49,316 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, vertexId=vertex_1463521969015_0001_1_00, initRequestedTime=1463521969255, initedTime=1463521969316, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 21:52:49,317 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] with distanceFromRoot: 0
2016-05-17 21:52:49,317 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 21:52:49,318 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,321 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:49,321 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,322 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,322 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,322 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=2A1A4FB267374160A082E625EEA6D434, vertexId=vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 21:52:49,323 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=2A1A4FB267374160A082E625EEA6D434, vertexId=vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,325 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=2A1A4FB267374160A082E625EEA6D434, vertexName=D04CA917C2D045518CC4DFBB4CC488FA, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 21:52:49,326 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=2A1A4FB267374160A082E625EEA6D434, vertexId=vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,327 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] submitted tez dag to app master: application_1463521969015_0001, with dag id: dag_1463521969015_0001_1
2016-05-17 21:52:49,327 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, vertexId=vertex_1463521969015_0001_1_03, initRequestedTime=1463521969317, initedTime=1463521969327, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:49,327 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:49,329 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:49,329 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,329 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,329 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:49,329 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521969015_0001_1_00, startRequestedTime=1463521969329, startedTime=1463521969329
2016-05-17 21:52:49,330 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:49,330 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:49,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_STARTED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, taskId=task_1463521969015_0001_1_00_000000, scheduledTime=1463521969330, launchTime=1463521969330
2016-05-17 21:52:49,331 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:49,331 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521969015_0001_1_00 for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] numStartedSources: 1 numSources: 1
2016-05-17 21:52:49,331 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] with distanceFromRoot: 1
2016-05-17 21:52:49,332 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,335 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B6FC4A96F0D14AD8A08DCB78D2443B04 in vertex: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,335 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C302CF9CA9EE441089950BB9BF36060E with 1 source tasks and 1 pending tasks
2016-05-17 21:52:49,335 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521969015_0001_1_01, startRequestedTime=1463521969335, startedTime=1463521969335
2016-05-17 21:52:49,335 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:49,336 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:49,337 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: B6FC4A96F0D14AD8A08DCB78D2443B04, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=1DFCDDDFA1784698BFBDD8A3F6120887, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,338 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:49,338 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521969015_0001_1_01 for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] numStartedSources: 1 numSources: 1
2016-05-17 21:52:49,338 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] with distanceFromRoot: 2
2016-05-17 21:52:49,339 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:49,340 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C302CF9CA9EE441089950BB9BF36060E in vertex: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,341 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 21:52:49,341 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DCDE984A0B164F5680D582A8CC0575C9 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:49,342 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:49,342 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521969015_0001_1_02, startRequestedTime=1463521969340, startedTime=1463521969340
2016-05-17 21:52:49,342 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:49,343 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 21:52:49,343 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 21:52:49,344 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521969015_0001_00_000001
2016-05-17 21:52:49,344 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:49,345 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521969015_0001_1_02 for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] numStartedSources: 1 numSources: 1
2016-05-17 21:52:49,345 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] with distanceFromRoot: 3
2016-05-17 21:52:49,347 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: DCDE984A0B164F5680D582A8CC0575C9 in vertex: D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,347 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521969015_0001_00_000001
2016-05-17 21:52:49,348 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 21:52:49,348 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D04CA917C2D045518CC4DFBB4CC488FA with 1 source tasks and 1 pending tasks
2016-05-17 21:52:49,348 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:49,348 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521969015_0001_1_03, startRequestedTime=1463521969347, startedTime=1463521969347
2016-05-17 21:52:49,349 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:49,350 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:49,350 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521969015_0001_00_000001, launchTime=1463521969350
2016-05-17 21:52:49,350 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:49,350 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000001
2016-05-17 21:52:49,351 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521969015_0001_1_00_000000_0] to container: [container_1463521969015_0001_00_000001]
2016-05-17 21:52:49,352 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:49,352 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521969015_0001_00_000001 given task: attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,352 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,352 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521969015_0001_1_00_000000_0] started. Is using containerId: [container_1463521969015_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 21:52:49 Starting to run new task attempt: attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,353 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,353 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,353 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0, startTime=1463521969352, containerId=container_1463521969015_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521969015_0001_00_000001/teamcity, completedLogs=
2016-05-17 21:52:49,353 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:49,353 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:49,353 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:49,354 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:49,353 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:49,354 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:49,354 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:49,354 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:49,354 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: B6FC4A96F0D14AD8A08DCB78D2443B04, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=1DFCDDDFA1784698BFBDD8A3F6120887, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,355 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:49,356 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,358 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:49,359 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521969015_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 21:52:49,359 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,362 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:49,362 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=1DFCDDDFA1784698BFBDD8A3F6120887, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 21:52:49,363 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:49,363 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,364 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:49,363 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:49,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,367 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,367 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:49,368 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,368 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,371 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 21:52:49,371 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,371 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,379 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B6FC4A96F0D14AD8A08DCB78D2443B04, ordinal: 0
2016-05-17 21:52:49,379 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B6FC4A96F0D14AD8A08DCB78D2443B04, mem on start (mb), free: 247, total: 515, max: 672
2016-05-17 21:52:49,380 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,380 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:49,380 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:49,380 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:49,381 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:49,381 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 21:52:49,381 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:49,382 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:49,382 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:49,382 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 21:52:49,383 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 1DFCDDDFA1784698BFBDD8A3F6120887, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 21:52:49,383 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C302CF9CA9EE441089950BB9BF36060E, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 21:52:49,383 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:49,383 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 1DFCDDDFA1784698BFBDD8A3F6120887 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:49,384 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:49,384 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:49,383 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,385 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:49,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:49,385 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,386 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 21:52:49,386 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@4c7b6152
2016-05-17 21:52:49,388 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@4c7b6152
2016-05-17 21:52:49,388 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 21:52:49,388 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 21:52:49,390 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:49,391 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 1DFCDDDFA1784698BFBDD8A3F6120887
2016-05-17 21:52:49,391 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: 93BB685E5B4B4F10A625421CA2BE4012
2016-05-17 21:52:49,391 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 93BB685E5B4B4F10A625421CA2BE4012
2016-05-17 21:52:49,395 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@ea8c62a; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@6a69f6ea; comparator=cascading.tuple.hadoop.util.TupleComparator@52d1b33a; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:49,395 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:49,396 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:49,396 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,412 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 21:52:49,412 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 21:52:49,413 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 21:52:49,413 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 21:52:49,413 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 21:52:49,414 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:49,414 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:49,415 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B6FC4A96F0D14AD8A08DCB78D2443B04, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:49,421 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,421 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B6FC4A96F0D14AD8A08DCB78D2443B04, mem on close (mb), free: 144, total: 515, max: 672
2016-05-17 21:52:49,422 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:49,422 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:49,422 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-05-17 21:52:49,422 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-05-17 21:52:49,427 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:49 Completed running task attempt: attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,432 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:49,433 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521969015_0001_1_00, eventCount=1
2016-05-17 21:52:49,434 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:49,435 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, taskAttemptId=attempt_1463521969015_0001_1_00_000000_0, startTime=1463521969352, finishTime=1463521969434, timeTaken=82, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969415, Process_Duration=6, Process_End_Time=1463521969421, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:49,435 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:49,435 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:49,436 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463521969415, Process_Duration=6, Process_End_Time=1463521969421, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-17 21:52:49,435 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521969015_0001_1_00_000000_0
2016-05-17 21:52:49,437 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:49,437 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,437 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000001
2016-05-17 21:52:49,438 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_FINISHED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, taskId=task_1463521969015_0001_1_00_000000, startTime=1463521969352, finishTime=1463521969437, timeTaken=85, status=SUCCEEDED, successfulAttemptID=attempt_1463521969015_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969415, Process_Duration=6, Process_End_Time=1463521969421, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:49,438 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:49,438 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] : 1
2016-05-17 21:52:49,439 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,439 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:49,441 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:49,442 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_FINISHED]: vertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, vertexId=vertex_1463521969015_0001_1_00, initRequestedTime=1463521969255, initedTime=1463521969316, startRequestedTime=1463521969329, startedTime=1463521969329, finishTime=1463521969439, timeTaken=110, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969415, Process_Duration=6, Process_End_Time=1463521969421, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463521969352, firstTasksToStart=[ task_1463521969015_0001_1_00_000000 ], lastTaskFinishTime=1463521969434, lastTasksToFinish=[ task_1463521969015_0001_1_00_000000 ], minTaskDuration=82, maxTaskDuration=82, avgTaskDuration=82.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521969015_0001_1_00_000000 ], longestDurationTasks=[ task_1463521969015_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:49,442 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:49,443 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:49,443 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521969015_0001_00_000001, stoppedTime=1463521969443, exitStatus=0
2016-05-17 21:52:49,443 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:49,443 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:49,444 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] attempt: attempt_1463521969015_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:49,444 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,444 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,445 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:49,445 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521969015_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 21:52:49,446 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_STARTED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, taskId=task_1463521969015_0001_1_01_000000, scheduledTime=1463521969446, launchTime=1463521969446
2016-05-17 21:52:49,446 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:49,446 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:49,446 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:49,447 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,447 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: C302CF9CA9EE441089950BB9BF36060E, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,447 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:49,448 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:49,448 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521969015_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 21:52:49,449 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:49,449 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521969015_0001_00_000001, Exiting
2016-05-17 21:52:49,449 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521969015_0001_00_000001
2016-05-17 21:52:49,449 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521969015_0001_00_000001
2016-05-17 21:52:49,448 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521969015_0001_00_000002
2016-05-17 21:52:49,450 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:49,452 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521969015_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:49,453 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521969015_0001_00_000001 completed successfully
2016-05-17 21:52:49,454 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521969015_0001_00_000002
2016-05-17 21:52:49,455 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:49,455 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:49,457 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:49,457 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:49,457 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521969015_0001_00_000002, launchTime=1463521969457
2016-05-17 21:52:49,458 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000002
2016-05-17 21:52:49,459 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521969015_0001_1_01_000000_0] to container: [container_1463521969015_0001_00_000002]
2016-05-17 21:52:49,459 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:49,459 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521969015_0001_1_01_000000_0] started. Is using containerId: [container_1463521969015_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 21:52:49,460 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0, startTime=1463521969459, containerId=container_1463521969015_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521969015_0001_00_000002/teamcity, completedLogs=
2016-05-17 21:52:49,460 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:49,460 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:49,460 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:49,460 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521969015_0001_00_000002 given task: attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,460 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49 Starting to run new task attempt: attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,461 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,461 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,462 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:49,462 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:49,463 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:49,463 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:49,463 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:49,463 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: C302CF9CA9EE441089950BB9BF36060E, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,464 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:49,464 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:49,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,465 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521969015_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 21:52:49,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B6FC4A96F0D14AD8A08DCB78D2443B04, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:49,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:49,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:49,469 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,470 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:49,470 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,471 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:49,471 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:49,471 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,471 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,472 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:49,472 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,474 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:49,474 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,483 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C302CF9CA9EE441089950BB9BF36060E, ordinal: 1
2016-05-17 21:52:49,483 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C302CF9CA9EE441089950BB9BF36060E, mem on start (mb), free: 138, total: 515, max: 672
2016-05-17 21:52:49,483 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,483 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:49,483 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:49,484 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:49,484 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:49,484 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:49,484 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:49,485 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:49,485 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:49,485 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B6FC4A96F0D14AD8A08DCB78D2443B04, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:49,485 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DCDE984A0B164F5680D582A8CC0575C9, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:49,486 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:49,486 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B6FC4A96F0D14AD8A08DCB78D2443B04 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:49,486 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,487 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:49,488 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:49,488 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,490 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:49,491 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B6FC4A96F0D14AD8A08DCB78D2443B04 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:49,491 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:49,491 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:49,492 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:49,493 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,493 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:49,495 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,495 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521969015_0001_1_00_000000_0_10014, runDuration: 65985, ]
2016-05-17 21:52:49,497 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521969015_0001_1_00_000000_0_10014] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1463521969497, TimeTaken=1, Rate=1.04 MB/s
2016-05-17 21:52:49,498 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,498 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:49,499 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [B6FC4A96F0D14AD8A08DCB78D2443B04] #4 in 3ms
2016-05-17 21:52:49,499 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-05-17 21:52:49,499 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B6FC4A96F0D14AD8A08DCB78D2443B04] #4, status:false, isInterrupted:false
2016-05-17 21:52:49,500 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:49,501 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 93BB685E5B4B4F10A625421CA2BE4012
2016-05-17 21:52:49,501 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-05-17 21:52:49,501 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,501 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:49,502 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-17 21:52:49,502 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,502 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:49,502 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: 3AFC71E4C0AB4E0EB267E9AB30477318
2016-05-17 21:52:49,503 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] 3AFC71E4C0AB4E0EB267E9AB30477318
2016-05-17 21:52:49,506 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@47c1dd34; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@8a1a5ba; comparator=cascading.tuple.hadoop.util.TupleComparator@2c875ac3; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:49,507 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:49,507 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:49,507 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,570 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:49,570 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:49,571 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:49,571 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:49,571 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:49,571 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 93BB685E5B4B4F10A625421CA2BE4012, for 1 inputs
2016-05-17 21:52:49,574 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:49,575 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:49,576 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C302CF9CA9EE441089950BB9BF36060E, all 1 inputs ready in: 00:00:00.001
2016-05-17 21:52:49,578 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@5a9bfc60; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@36d63119
2016-05-17 21:52:49,609 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,610 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C302CF9CA9EE441089950BB9BF36060E, mem on close (mb), free: 69, total: 515, max: 672
2016-05-17 21:52:49,610 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B6FC4A96F0D14AD8A08DCB78D2443B04
2016-05-17 21:52:49,610 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:49,611 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:49,612 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,612 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:49,612 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:49,612 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:49,613 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:49,613 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-05-17 21:52:49,613 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-05-17 21:52:49,634 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:49 Completed running task attempt: attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,640 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:49,640 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521969015_0001_1_01, eventCount=1
2016-05-17 21:52:49,641 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521969576, Process_Duration=33, Process_End_Time=1463521969609, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-05-17 21:52:49,642 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:49,642 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,643 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:49,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, taskAttemptId=attempt_1463521969015_0001_1_01_000000_0, startTime=1463521969459, finishTime=1463521969642, timeTaken=183, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969576, Process_Duration=33, Process_End_Time=1463521969609, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-05-17 21:52:49,643 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:49,644 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521969015_0001_1_01_000000_0
2016-05-17 21:52:49,644 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:49,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_FINISHED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, taskId=task_1463521969015_0001_1_01_000000, startTime=1463521969459, finishTime=1463521969644, timeTaken=185, status=SUCCEEDED, successfulAttemptID=attempt_1463521969015_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969576, Process_Duration=33, Process_End_Time=1463521969609, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-05-17 21:52:49,645 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:49,645 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:49,646 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] : 1
2016-05-17 21:52:49,646 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,646 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000002
2016-05-17 21:52:49,648 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:49,649 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_FINISHED]: vertexName=C302CF9CA9EE441089950BB9BF36060E, vertexId=vertex_1463521969015_0001_1_01, initRequestedTime=1463521969257, initedTime=1463521969265, startRequestedTime=1463521969335, startedTime=1463521969335, finishTime=1463521969646, timeTaken=311, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969576, Process_Duration=33, Process_End_Time=1463521969609, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1463521969459, firstTasksToStart=[ task_1463521969015_0001_1_01_000000 ], lastTaskFinishTime=1463521969642, lastTasksToFinish=[ task_1463521969015_0001_1_01_000000 ], minTaskDuration=183, maxTaskDuration=183, avgTaskDuration=183.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521969015_0001_1_01_000000 ], longestDurationTasks=[ task_1463521969015_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:49,649 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:49,649 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:49,649 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] attempt: attempt_1463521969015_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:49,650 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,650 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,650 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:49,650 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521969015_0001_00_000002, stoppedTime=1463521969650, exitStatus=0
2016-05-17 21:52:49,651 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:49,651 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:49,652 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:49,652 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_STARTED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, taskId=task_1463521969015_0001_1_02_000000, scheduledTime=1463521969651, launchTime=1463521969651
2016-05-17 21:52:49,652 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:49,652 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521969015_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-17 21:52:49,652 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,653 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:49,653 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: DCDE984A0B164F5680D582A8CC0575C9, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D04CA917C2D045518CC4DFBB4CC488FA, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,654 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:49,654 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:49,655 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521969015_0001_00_000003
2016-05-17 21:52:49,655 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:49,657 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521969015_0001_00_000003
2016-05-17 21:52:49,657 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-05-17 21:52:49,657 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521969015_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 21:52:49,657 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:49,657 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521969015_0001_00_000002, Exiting
2016-05-17 21:52:49,657 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521969015_0001_00_000002
2016-05-17 21:52:49,658 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521969015_0001_00_000002
2016-05-17 21:52:49,657 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:49,658 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:49,659 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521969015_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:49,659 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521969015_0001_00_000002 completed successfully
2016-05-17 21:52:49,658 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521969015_0001_00_000003, launchTime=1463521969657
2016-05-17 21:52:49,659 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:49,659 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000003
2016-05-17 21:52:49,660 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521969015_0001_1_02_000000_0] to container: [container_1463521969015_0001_00_000003]
2016-05-17 21:52:49,660 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:49,660 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521969015_0001_00_000003 given task: attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,660 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,661 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521969015_0001_1_02_000000_0] started. Is using containerId: [container_1463521969015_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-17 21:52:49 Starting to run new task attempt: attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,661 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,661 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,661 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:49,661 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:49,661 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0, startTime=1463521969660, containerId=container_1463521969015_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521969015_0001_00_000003/teamcity, completedLogs=
2016-05-17 21:52:49,661 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:49,662 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:49,662 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:49,662 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:49,662 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:49,663 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: DCDE984A0B164F5680D582A8CC0575C9, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D04CA917C2D045518CC4DFBB4CC488FA, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:49,662 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:49,663 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:49,663 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,663 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:49,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,664 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521969015_0001_1_02_000000_0 sent events: (0-1)
2016-05-17 21:52:49,667 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C302CF9CA9EE441089950BB9BF36060E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:49,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:49,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,671 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:49,671 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:49,671 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,672 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D04CA917C2D045518CC4DFBB4CC488FA, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:49,673 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:49,673 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:49,673 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,676 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:49,677 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,686 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DCDE984A0B164F5680D582A8CC0575C9, ordinal: 2
2016-05-17 21:52:49,686 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DCDE984A0B164F5680D582A8CC0575C9, mem on start (mb), free: 62, total: 515, max: 672
2016-05-17 21:52:49,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:49,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:49,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:49,687 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:49,687 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:49,687 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:49,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:49,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:49,688 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C302CF9CA9EE441089950BB9BF36060E, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:49,689 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D04CA917C2D045518CC4DFBB4CC488FA, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:49,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:49,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C302CF9CA9EE441089950BB9BF36060E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:49,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:49,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:49,691 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,692 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:49,692 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C302CF9CA9EE441089950BB9BF36060E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:49,692 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:49,693 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:49,693 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:49,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,694 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:49,695 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,699 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-05-17 21:52:49,700 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: 3AFC71E4C0AB4E0EB267E9AB30477318
2016-05-17 21:52:49,700 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: 34AE0CAB096F47D3BE48445ABB3463AA
2016-05-17 21:52:49,700 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 34AE0CAB096F47D3BE48445ABB3463AA
2016-05-17 21:52:49,700 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521969015_0001_1_01_000000_0_10018, runDuration: 167238, ]
2016-05-17 21:52:49,703 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@41a68efc; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@1df56410; comparator=cascading.tuple.hadoop.util.TupleComparator@75ffa3a0; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:49,703 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:49,703 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:49,704 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,735 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:49,735 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:49,735 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521969015_0001_1_01_000000_0_10018] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1463521969735, TimeTaken=33, Rate=0.22 MB/s
2016-05-17 21:52:49,736 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,736 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-17 21:52:49,736 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [C302CF9CA9EE441089950BB9BF36060E] #5 in 34ms
2016-05-17 21:52:49,736 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:49,736 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:49,737 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:49,737 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C302CF9CA9EE441089950BB9BF36060E] #5, status:false, isInterrupted:false
2016-05-17 21:52:49,738 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:49,738 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-05-17 21:52:49,738 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,739 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:49,739 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-17 21:52:49,739 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,740 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:49,740 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] 3AFC71E4C0AB4E0EB267E9AB30477318, for 1 inputs
2016-05-17 21:52:49,740 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DCDE984A0B164F5680D582A8CC0575C9, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:49,743 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@990e63; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@407f85e3
2016-05-17 21:52:49,767 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,767 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DCDE984A0B164F5680D582A8CC0575C9, mem on close (mb), free: 200, total: 516, max: 672
2016-05-17 21:52:49,767 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C302CF9CA9EE441089950BB9BF36060E
2016-05-17 21:52:49,768 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:49,769 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:49,769 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,769 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:49,769 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:49,770 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:49,770 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:49,771 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-05-17 21:52:49,771 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-05-17 21:52:49,776 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:49 Completed running task attempt: attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,780 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:49,782 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521969015_0001_1_02, eventCount=1
2016-05-17 21:52:49,782 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:49,783 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, taskAttemptId=attempt_1463521969015_0001_1_02_000000_0, startTime=1463521969660, finishTime=1463521969782, timeTaken=122, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4896, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=47, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969741, Process_Duration=26, Process_End_Time=1463521969767, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-05-17 21:52:49,783 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:49,783 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:49,783 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521969015_0001_1_02_000000_0
2016-05-17 21:52:49,784 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:49,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_FINISHED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, taskId=task_1463521969015_0001_1_02_000000, startTime=1463521969660, finishTime=1463521969784, timeTaken=124, status=SUCCEEDED, successfulAttemptID=attempt_1463521969015_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4896, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=47, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969741, Process_Duration=26, Process_End_Time=1463521969767, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-05-17 21:52:49,785 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:49,785 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] : 1
2016-05-17 21:52:49,785 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4896, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=47, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521969741, Process_Duration=26, Process_End_Time=1463521969767, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-05-17 21:52:49,786 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:49,786 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,787 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000003
2016-05-17 21:52:49,787 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:49,787 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_FINISHED]: vertexName=DCDE984A0B164F5680D582A8CC0575C9, vertexId=vertex_1463521969015_0001_1_02, initRequestedTime=1463521969266, initedTime=1463521969312, startRequestedTime=1463521969340, startedTime=1463521969340, finishTime=1463521969785, timeTaken=445, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4896, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=47, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969741, Process_Duration=26, Process_End_Time=1463521969767, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1463521969660, firstTasksToStart=[ task_1463521969015_0001_1_02_000000 ], lastTaskFinishTime=1463521969782, lastTasksToFinish=[ task_1463521969015_0001_1_02_000000 ], minTaskDuration=122, maxTaskDuration=122, avgTaskDuration=122.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521969015_0001_1_02_000000 ], longestDurationTasks=[ task_1463521969015_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:49,788 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:49,788 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:49,788 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521969015_0001_00_000003, stoppedTime=1463521969788, exitStatus=0
2016-05-17 21:52:49,788 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:49,788 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:49,788 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] attempt: attempt_1463521969015_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:49,789 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D04CA917C2D045518CC4DFBB4CC488FA
2016-05-17 21:52:49,789 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,789 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:49,789 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521969015_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-17 21:52:49,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_STARTED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, taskId=task_1463521969015_0001_1_03_000000, scheduledTime=1463521969789, launchTime=1463521969789
2016-05-17 21:52:49,790 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:49,790 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:49,791 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,789 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:49,791 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: D04CA917C2D045518CC4DFBB4CC488FA, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=2A1A4FB267374160A082E625EEA6D434, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:49,792 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:49,792 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:49,793 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521969015_0001_00_000004
2016-05-17 21:52:49,794 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:49,796 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521969015_0001_00_000004
2016-05-17 21:52:49,797 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:49,797 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521969015_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-17 21:52:49,797 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000003: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:49,797 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521969015_0001_00_000003, Exiting
2016-05-17 21:52:49,798 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521969015_0001_00_000003
2016-05-17 21:52:49,798 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521969015_0001_00_000003
2016-05-17 21:52:49,800 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521969015_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:49,800 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521969015_0001_00_000004, launchTime=1463521969800
2016-05-17 21:52:49,800 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:49,800 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521969015_0001_00_000003 completed successfully
2016-05-17 21:52:49,801 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:49,801 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000004
2016-05-17 21:52:49,801 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:49,802 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521969015_0001_1_03_000000_0] to container: [container_1463521969015_0001_00_000004]
2016-05-17 21:52:49,802 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:49,802 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521969015_0001_00_000004 given task: attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,803 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49 Starting to run new task attempt: attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,803 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521969015_0001_1_03_000000_0] started. Is using containerId: [container_1463521969015_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-17 21:52:49,803 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,803 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0, startTime=1463521969803, containerId=container_1463521969015_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521969015_0001_00_000004/teamcity, completedLogs=
2016-05-17 21:52:49,804 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:49,804 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:49,803 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:49,804 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:49,803 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:49,804 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:49,805 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:49,805 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:49,805 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16] pearson/(1/1) ...mposite/pearson/composite, VertexName: D04CA917C2D045518CC4DFBB4CC488FA, VertexParallelism: 1, TaskAttemptID:attempt_1463521969015_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=2A1A4FB267374160A082E625EEA6D434, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:49,805 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:49,808 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521969015_0001_1_03_000000_0 sent events: (0-1)
2016-05-17 21:52:49,809 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:49,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=2A1A4FB267374160A082E625EEA6D434, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 21:52:49,812 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:49,812 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 2A1A4FB267374160A082E625EEA6D434
2016-05-17 21:52:49,813 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 21:52:49,813 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 2A1A4FB267374160A082E625EEA6D434
2016-05-17 21:52:49,814 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:49,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DCDE984A0B164F5680D582A8CC0575C9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:49,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:49,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,821 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:49,821 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:49,822 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,824 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,829 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D04CA917C2D045518CC4DFBB4CC488FA, ordinal: 3
2016-05-17 21:52:49,830 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D04CA917C2D045518CC4DFBB4CC488FA, mem on start (mb), free: 192, total: 516, max: 672
2016-05-17 21:52:49,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:49,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:49,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:49,833 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 21:52:49,833 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 2A1A4FB267374160A082E625EEA6D434
2016-05-17 21:52:49,833 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:49,834 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:49,834 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 21:52:49,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:49,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:49,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:49,836 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 21:52:49,836 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 2A1A4FB267374160A082E625EEA6D434, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 21:52:49,836 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DCDE984A0B164F5680D582A8CC0575C9, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 21:52:49,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:49,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DCDE984A0B164F5680D582A8CC0575C9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:49,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:49,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:49,838 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:49,839 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:49,839 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: DCDE984A0B164F5680D582A8CC0575C9 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:49,839 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 21:52:49,839 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:49,840 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:49,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:49,841 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,841 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521969015_0001_1_02_000000_0_10021, runDuration: 105014, ]
2016-05-17 21:52:49,844 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-05-17 21:52:49,845 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521969015_0001_1_02_000000_0_10021] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1463521969844, TimeTaken=2, Rate=2.07 MB/s
2016-05-17 21:52:49,845 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: 34AE0CAB096F47D3BE48445ABB3463AA
2016-05-17 21:52:49,846 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: 2A1A4FB267374160A082E625EEA6D434
2016-05-17 21:52:49,847 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-05-17 21:52:49,848 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 34AE0CAB096F47D3BE48445ABB3463AA, for 1 inputs
2016-05-17 21:52:49,848 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,848 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:49,849 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [DCDE984A0B164F5680D582A8CC0575C9] #6 in 7ms
2016-05-17 21:52:49,849 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [DCDE984A0B164F5680D582A8CC0575C9] #6, status:false, isInterrupted:false
2016-05-17 21:52:49,850 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:49,850 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-05-17 21:52:49,850 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,850 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:49,851 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-05-17 21:52:49,851 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,851 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:49,852 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D04CA917C2D045518CC4DFBB4CC488FA, all 1 inputs ready in: 00:00:00.004
2016-05-17 21:52:49,854 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@14d97f8d; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@53450cac
2016-05-17 21:52:49,874 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 21:52:49,874 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 21:52:49,875 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,875 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463521969015_0001_1_03_000000_0 given a go for committing the task output.
2016-05-17 21:52:49,876 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,876 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D04CA917C2D045518CC4DFBB4CC488FA, mem on close (mb), free: 189, total: 516, max: 672
2016-05-17 21:52:49,877 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: DCDE984A0B164F5680D582A8CC0575C9
2016-05-17 21:52:49,877 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:49,878 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:49,878 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:49,878 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:49,879 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:49 Completed running task attempt: attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,880 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:49,881 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:49,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, taskAttemptId=attempt_1463521969015_0001_1_03_000000_0, startTime=1463521969803, finishTime=1463521969881, timeTaken=78, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969852, Process_Duration=24, Process_End_Time=1463521969876, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=8, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:49,882 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521969015_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:49,882 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521969015_0001_1_03_000000_0
2016-05-17 21:52:49,882 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:49,883 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:49,884 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:TASK_FINISHED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, taskId=task_1463521969015_0001_1_03_000000, startTime=1463521969803, finishTime=1463521969882, timeTaken=79, status=SUCCEEDED, successfulAttemptID=attempt_1463521969015_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969852, Process_Duration=24, Process_End_Time=1463521969876, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=8, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:49,884 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521969015_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:49,884 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] : 1
2016-05-17 21:52:49,884 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,886 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:49,886 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:VERTEX_FINISHED]: vertexName=D04CA917C2D045518CC4DFBB4CC488FA, vertexId=vertex_1463521969015_0001_1_03, initRequestedTime=1463521969317, initedTime=1463521969327, startRequestedTime=1463521969347, startedTime=1463521969347, finishTime=1463521969884, timeTaken=537, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521969852, Process_Duration=24, Process_End_Time=1463521969876, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=8, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463521969803, firstTasksToStart=[ task_1463521969015_0001_1_03_000000 ], lastTaskFinishTime=1463521969881, lastTasksToFinish=[ task_1463521969015_0001_1_03_000000 ], minTaskDuration=78, maxTaskDuration=78, avgTaskDuration=78.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521969015_0001_1_03_000000 ], longestDurationTasks=[ task_1463521969015_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:49,886 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:49,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521969852, Process_Duration=24, Process_End_Time=1463521969876, Read_Duration=4, Tuples_Read=87, Tuples_Written=21, Write_Duration=8][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-17 21:52:49,887 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:49,888 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:49,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521969015_0001_00_000004, stoppedTime=1463521969887, exitStatus=0
2016-05-17 21:52:49,888 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:49,888 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:49,889 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463521969015_0001_1
2016-05-17 21:52:49,889 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:49,889 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,890 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521969015_0001_00_000004
2016-05-17 21:52:49,890 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521969015_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-17 21:52:49,890 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521969015_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:49,890 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521969015_0001_00_000004, Exiting
2016-05-17 21:52:49,890 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521969015_0001_00_000004
2016-05-17 21:52:49,891 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521969015_0001_00_000004
2016-05-17 21:52:49,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463521969015_0001_1
2016-05-17 21:52:49,891 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521969015_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:49,891 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521969015_0001_00_000004 completed successfully
2016-05-17 21:52:49,891 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 2A1A4FB267374160A082E625EEA6D434 for vertex: vertex_1463521969015_0001_1_03 [D04CA917C2D045518CC4DFBB4CC488FA]
2016-05-17 21:52:49,901 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521969015_0001_1_02 [DCDE984A0B164F5680D582A8CC0575C9]
2016-05-17 21:52:49,901 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521969015_0001_1_01 [C302CF9CA9EE441089950BB9BF36060E]
2016-05-17 21:52:49,901 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521969015_0001_1_00 [B6FC4A96F0D14AD8A08DCB78D2443B04]
2016-05-17 21:52:49,905 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463521969015_0001_1, queueSize=0
2016-05-17 21:52:49,906 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 21:52:49,906 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521969015_0001_1][Event:DAG_FINISHED]: dagId=dag_1463521969015_0001_1, startTime=1463521969254, finishTime=1463521969901, timeTaken=647, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=15457, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=23, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2164260864, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=64, MERGE_PHASE_TIME=71, FIRST_EVENT_RECEIVED=18, LAST_EVENT_RECEIVED=18, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854087878584, Process_Duration=89, Process_End_Time=5854087878673, Read_Duration=18, Tuples_Read=303, Tuples_Written=317, Write_Duration=18, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-17 21:52:49,906 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463521969015_0001_1 finished with state: SUCCEEDED
2016-05-17 21:52:49,906 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521969015_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 21:52:49,907 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521969015_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:49,907 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521969015_0001_00_000004
2016-05-17 21:52:49 Completed Dag: dag_1463521969015_0001_1
2016-05-17 21:52:49,908 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463521969015_0001_1, dagState=SUCCEEDED
2016-05-17 21:52:49,908 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521969015_0001_1_post
2016-05-17 21:52:49,908 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:49,908 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 21:52:49,915 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463521969015_0001
2016-05-17 21:52:49,918 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 21:52:49,918 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 21:52:49,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 21:52:49,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 21:52:49,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 21:52:49,919 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 21:52:49,919 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 21:52:49,920 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 21:52:49,920 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 21:52:49,920 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 21:52:49,920 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 21:52:49,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 21:52:49,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 21:52:49,927 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/EFF69DD061E643EDAE65E1B32A2EB392/D2F21CBF2C1D4F4C9025AB3D17DDDA16/.tez/application_1463521969015_0001
2016-05-17 21:52:49,927 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite/_temporary
2016-05-17 21:52:49,928 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] completed in: 00:00.938
2016-05-17 21:52:50,007 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-05-17 21:52:50,053 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 21:52:50,054 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 21:52:50,137 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.051
2016-05-17 21:52:50,197 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.111
2016-05-17 21:52:50,198 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:50,198 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:50,199 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 21:52:50,231 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-05-17 21:52:50,268 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-05-17 21:52:50,421 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-17 21:52:50,422 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:50,422 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-05-17 21:52:50,422 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-17 21:52:50,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-17 21:52:50,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-17 21:52:50,424 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-05-17 21:52:50,424 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 21:52:50,439 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-17 21:52:50,445 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:50,446 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 21:52:50,446 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 21:52:50,456 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001 doesn't exist and is created
2016-05-17 21:52:50,475 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 21:52:50,475 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:50,476 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001
2016-05-17 21:52:50,481 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463521970445_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:50,493 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 21:52:50,494 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 21:52:50,496 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 21:52:50,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 21:52:50,497 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001/localmode-log-dir
2016-05-17 21:52:50,497 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001/localmode-log-dir/history.txt.appattempt_1463521970445_0001_000000, maxErrors=10
2016-05-17 21:52:50,497 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 21:52:50,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463521970445_0001_000000, appSubmitTime=1463521970480, launchTime=1463521970481
2016-05-17 21:52:50,506 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 21:52:50,507 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 21:52:50,509 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 21:52:50,509 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 21:52:50,511 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52801
2016-05-17 21:52:50,512 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 21:52:50,515 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463521970445_0001_000000, startTime=1463521970515
2016-05-17 21:52:50,515 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 21:52:50,515 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 21:52:50,576 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 21:52:50,576 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 21:52:50,576 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463521970445_0001, dagName=[C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-17 21:52:50,595 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-17 21:52:50,597 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463521970445_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001/localmode-log-dir/dag_1463521970445_0001_1.dot
2016-05-17 21:52:50,599 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001/localmode-log-dir/dag_1463521970445_0001_1-tez-dag.pb.txt
2016-05-17 21:52:50,640 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521970445_0001_1
2016-05-17 21:52:50,640 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,641 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-17 21:52:50 Running Dag: dag_1463521970445_0001_1
2016-05-17 21:52:50,658 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 21:52:50,659 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463521970445_0001_1, submitTime=1463521970596
2016-05-17 21:52:50,660 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:50,661 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 21:52:50,661 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 21:52:50,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463521970445_0001_1, initTime=1463521970659
2016-05-17 21:52:50,662 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521970445_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 21:52:50,663 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:DAG_STARTED]: dagID=dag_1463521970445_0001_1, startTime=1463521970663
2016-05-17 21:52:50,664 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 21:52:50,664 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521970445_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 21:52:50,664 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463521970445_0001, dagName=[C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-17 21:52:50,664 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: D648923925F143B998A537205192CD1A : {E886BC7C1BD14C7B816FDCDC99A80636={InputName=E886BC7C1BD14C7B816FDCDC99A80636}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 21:52:50,664 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: E886BC7C1BD14C7B816FDCDC99A80636, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 21:52:50,664 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,664 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,665 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,665 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]. Starting root input initializers: 1
2016-05-17 21:52:50,665 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 21:52:50,666 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 21:52:50,666 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,666 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: E886BC7C1BD14C7B816FDCDC99A80636 on vertex vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,669 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 21:52:50,670 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input E886BC7C1BD14C7B816FDCDC99A80636 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 21:52:50,671 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 21:52:50,671 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 21:52:50,672 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:50,673 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,674 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,674 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:50,675 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, vertexId=vertex_1463521970445_0001_1_01, initRequestedTime=1463521970666, initedTime=1463521970674, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:50,675 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:50,675 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:50,676 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-05-17 21:52:50,676 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: E886BC7C1BD14C7B816FDCDC99A80636 on vertex vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,679 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:50,679 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:50,680 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:50,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, vertexId=vertex_1463521970445_0001_1_02, initRequestedTime=1463521970675, initedTime=1463521970680, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:50,680 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:50,680 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:50,681 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:50,681 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {E886BC7C1BD14C7B816FDCDC99A80636=forAllWorkUnits=true, update=[1]}
2016-05-17 21:52:50,681 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] parallelism set to 1
2016-05-17 21:52:50,681 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:50,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521970445_0001_1_00, eventCount=1
2016-05-17 21:52:50,683 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:50,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D648923925F143B998A537205192CD1A, vertexId=vertex_1463521970445_0001_1_00, initRequestedTime=1463521970664, initedTime=1463521970682, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 21:52:50,684 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] with distanceFromRoot: 0
2016-05-17 21:52:50,684 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 21:52:50,684 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,690 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:50,691 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,691 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,691 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,691 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=C61A8119CF93417FAAD84646D126A048, vertexId=vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 21:52:50,692 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=C61A8119CF93417FAAD84646D126A048, vertexId=vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,694 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=C61A8119CF93417FAAD84646D126A048, vertexName=E521D1AA5A254346841DC9D7ED2FAA08, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 21:52:50,694 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=C61A8119CF93417FAAD84646D126A048, vertexId=vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:50,696 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:50,696 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, vertexId=vertex_1463521970445_0001_1_03, initRequestedTime=1463521970684, initedTime=1463521970696, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:50,696 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:50,696 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463521970445_0001, with dag id: dag_1463521970445_0001_1
2016-05-17 21:52:50,697 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,699 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521970445_0001_1_00, startRequestedTime=1463521970696, startedTime=1463521970696
2016-05-17 21:52:50,700 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:50,700 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:50,702 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:50,702 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_STARTED]: vertexName=D648923925F143B998A537205192CD1A, taskId=task_1463521970445_0001_1_00_000000, scheduledTime=1463521970701, launchTime=1463521970701
2016-05-17 21:52:50,703 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:50,703 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521970445_0001_1_00 for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] numStartedSources: 1 numSources: 1
2016-05-17 21:52:50,703 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] with distanceFromRoot: 1
2016-05-17 21:52:50,703 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,703 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D648923925F143B998A537205192CD1A in vertex: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,704 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: FC155BF00B274480B4AC9C6A873ACAC0 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:50,704 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:50,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521970445_0001_1_01, startRequestedTime=1463521970703, startedTime=1463521970703
2016-05-17 21:52:50,705 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:50,705 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D648923925F143B998A537205192CD1A, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E886BC7C1BD14C7B816FDCDC99A80636, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,706 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:50,706 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521970445_0001_1_01 for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] numStartedSources: 1 numSources: 1
2016-05-17 21:52:50,706 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] with distanceFromRoot: 2
2016-05-17 21:52:50,707 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:50,707 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: FC155BF00B274480B4AC9C6A873ACAC0 in vertex: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:50,708 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: EF13735CD4F94A68AA6A6B96259AE027 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:50,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521970445_0001_1_02, startRequestedTime=1463521970707, startedTime=1463521970707
2016-05-17 21:52:50,708 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:50,708 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:50,709 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521970445_0001_1_02 for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] numStartedSources: 1 numSources: 1
2016-05-17 21:52:50,709 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] with distanceFromRoot: 3
2016-05-17 21:52:50,709 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: EF13735CD4F94A68AA6A6B96259AE027 in vertex: E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:50,709 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E521D1AA5A254346841DC9D7ED2FAA08 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:50,710 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:50,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521970445_0001_1_03, startRequestedTime=1463521970709, startedTime=1463521970709
2016-05-17 21:52:50,710 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:50,710 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 21:52:50,712 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 21:52:50,713 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 21:52:50,714 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521970445_0001_00_000001
2016-05-17 21:52:50,714 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:50,717 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521970445_0001_00_000001
2016-05-17 21:52:50,717 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 21:52:50,718 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:50,718 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:50,718 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521970445_0001_00_000001, launchTime=1463521970718
2016-05-17 21:52:50,719 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000001
2016-05-17 21:52:50,720 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521970445_0001_1_00_000000_0] to container: [container_1463521970445_0001_00_000001]
2016-05-17 21:52:50,720 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:50,721 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521970445_0001_1_00_000000_0] started. Is using containerId: [container_1463521970445_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 21:52:50,721 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:50,721 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D648923925F143B998A537205192CD1A, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0, startTime=1463521970720, containerId=container_1463521970445_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521970445_0001_00_000001/teamcity, completedLogs=
2016-05-17 21:52:50,721 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:50,721 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521970445_0001_00_000001 given task: attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,721 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:50,721 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50 Starting to run new task attempt: attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,722 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,722 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,722 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:50,723 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:50,723 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:50,723 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:50,723 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:50,723 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D648923925F143B998A537205192CD1A, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E886BC7C1BD14C7B816FDCDC99A80636, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,724 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:50,724 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:50,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,725 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521970445_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 21:52:50,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E886BC7C1BD14C7B816FDCDC99A80636, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 21:52:50,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:50,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:50,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,727 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:50,730 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 21:52:50,731 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,737 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:50,737 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,739 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:50,746 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D648923925F143B998A537205192CD1A, ordinal: 0
2016-05-17 21:52:50,746 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D648923925F143B998A537205192CD1A, mem on start (mb), free: 51, total: 516, max: 672
2016-05-17 21:52:50,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:50,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:50,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:50,747 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:50,748 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 21:52:50,748 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:50,748 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:50,748 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:50,749 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 21:52:50,749 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E886BC7C1BD14C7B816FDCDC99A80636, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 21:52:50,749 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, FC155BF00B274480B4AC9C6A873ACAC0, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 21:52:50,749 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:50,750 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E886BC7C1BD14C7B816FDCDC99A80636 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:50,750 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:50,750 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:50,750 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,766 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:50,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:50,767 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,769 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 21:52:50,769 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@7e25a483
2016-05-17 21:52:50,771 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:50,771 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@7e25a483
2016-05-17 21:52:50,771 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 21:52:50,771 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 21:52:50,772 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: E886BC7C1BD14C7B816FDCDC99A80636
2016-05-17 21:52:50,772 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 96CE1E94E5834D08A72B45BAC6B0CA4F
2016-05-17 21:52:50,772 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 96CE1E94E5834D08A72B45BAC6B0CA4F
2016-05-17 21:52:50,774 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@7cf3d8b3; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@6d180fdb; comparator=cascading.tuple.hadoop.util.TupleComparator@46b92ad1; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:50,774 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:50,774 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:50,775 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:50,788 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 21:52:50,788 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 21:52:50,788 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 21:52:50,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 21:52:50,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 21:52:50,790 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:50,790 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:50,791 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D648923925F143B998A537205192CD1A, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:50,795 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,795 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D648923925F143B998A537205192CD1A, mem on close (mb), free: 160, total: 515, max: 672
2016-05-17 21:52:50,796 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:50,796 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:50,796 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-05-17 21:52:50,796 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-05-17 21:52:50,801 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:50 Completed running task attempt: attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,805 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:50,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521970445_0001_1_00, eventCount=1
2016-05-17 21:52:50,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=14, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463521970791, Process_Duration=4, Process_End_Time=1463521970795, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-17 21:52:50,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D648923925F143B998A537205192CD1A, taskAttemptId=attempt_1463521970445_0001_1_00_000000_0, startTime=1463521970720, finishTime=1463521970808, timeTaken=88, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=14, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970791, Process_Duration=4, Process_End_Time=1463521970795, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:50,808 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:50,810 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:50,810 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:50,810 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:50,811 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,811 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000001
2016-05-17 21:52:50,810 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521970445_0001_1_00_000000_0
2016-05-17 21:52:50,812 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:50,812 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_FINISHED]: vertexName=D648923925F143B998A537205192CD1A, taskId=task_1463521970445_0001_1_00_000000, startTime=1463521970720, finishTime=1463521970811, timeTaken=91, status=SUCCEEDED, successfulAttemptID=attempt_1463521970445_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=14, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970791, Process_Duration=4, Process_End_Time=1463521970795, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:50,812 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:50,813 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] : 1
2016-05-17 21:52:50,813 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:50,815 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:50,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_FINISHED]: vertexName=D648923925F143B998A537205192CD1A, vertexId=vertex_1463521970445_0001_1_00, initRequestedTime=1463521970664, initedTime=1463521970682, startRequestedTime=1463521970696, startedTime=1463521970696, finishTime=1463521970813, timeTaken=117, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=14, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970791, Process_Duration=4, Process_End_Time=1463521970795, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463521970720, firstTasksToStart=[ task_1463521970445_0001_1_00_000000 ], lastTaskFinishTime=1463521970808, lastTasksToFinish=[ task_1463521970445_0001_1_00_000000 ], minTaskDuration=88, maxTaskDuration=88, avgTaskDuration=88.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521970445_0001_1_00_000000 ], longestDurationTasks=[ task_1463521970445_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:50,816 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:50,816 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:50,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521970445_0001_00_000001, stoppedTime=1463521970816, exitStatus=0
2016-05-17 21:52:50,816 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:50,817 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:50,817 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] attempt: attempt_1463521970445_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:50,818 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,818 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,818 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:50,818 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521970445_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 21:52:50,818 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_STARTED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, taskId=task_1463521970445_0001_1_01_000000, scheduledTime=1463521970818, launchTime=1463521970818
2016-05-17 21:52:50,818 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:50,819 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:50,821 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,821 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:50,821 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FC155BF00B274480B4AC9C6A873ACAC0, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D648923925F143B998A537205192CD1A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,822 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521970445_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 21:52:50,822 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:50,822 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521970445_0001_00_000001, Exiting
2016-05-17 21:52:50,822 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521970445_0001_00_000001
2016-05-17 21:52:50,822 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521970445_0001_00_000001
2016-05-17 21:52:50,822 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:50,823 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:50,826 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521970445_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:50,826 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521970445_0001_00_000001 completed successfully
2016-05-17 21:52:50,826 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521970445_0001_00_000002
2016-05-17 21:52:50,827 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:50,829 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:50,829 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521970445_0001_00_000002
2016-05-17 21:52:50,830 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:50,834 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:50,835 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:50,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521970445_0001_00_000002, launchTime=1463521970834
2016-05-17 21:52:50,837 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000002
2016-05-17 21:52:50,838 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521970445_0001_1_01_000000_0] to container: [container_1463521970445_0001_00_000002]
2016-05-17 21:52:50,838 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:50,838 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521970445_0001_1_01_000000_0] started. Is using containerId: [container_1463521970445_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 21:52:50,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0, startTime=1463521970838, containerId=container_1463521970445_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521970445_0001_00_000002/teamcity, completedLogs=
2016-05-17 21:52:50,838 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:50,839 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:50,839 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:50,839 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521970445_0001_00_000002 given task: attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,840 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50 Starting to run new task attempt: attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,840 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,840 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,841 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:50,841 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:50,841 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:50,841 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:50,841 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:50,842 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FC155BF00B274480B4AC9C6A873ACAC0, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D648923925F143B998A537205192CD1A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,842 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:50,842 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:50,844 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521970445_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 21:52:50,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:50,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D648923925F143B998A537205192CD1A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:50,846 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:50,846 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:50,847 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:50,847 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,849 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:50,849 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:50,849 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:50,850 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:50,852 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:50,852 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:50,852 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,860 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FC155BF00B274480B4AC9C6A873ACAC0, ordinal: 1
2016-05-17 21:52:50,860 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC155BF00B274480B4AC9C6A873ACAC0, mem on start (mb), free: 153, total: 515, max: 672
2016-05-17 21:52:50,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:50,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:50,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:50,861 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:50,861 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:50,861 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:50,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:50,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:50,862 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, EF13735CD4F94A68AA6A6B96259AE027, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:50,863 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D648923925F143B998A537205192CD1A, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:50,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:50,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D648923925F143B998A537205192CD1A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:50,864 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:50,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:50,864 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,867 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:50,868 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:50,869 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D648923925F143B998A537205192CD1A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:50,869 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:50,869 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:50,869 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:50,870 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,870 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:50,871 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,871 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521970445_0001_1_00_000000_0_10027, runDuration: 71187, ]
2016-05-17 21:52:50,873 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521970445_0001_1_00_000000_0_10027] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1463521970873, TimeTaken=0, Rate=0.00 MB/s
2016-05-17 21:52:50,873 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,873 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:50,873 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [D648923925F143B998A537205192CD1A] #7 in 2ms
2016-05-17 21:52:50,873 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D648923925F143B998A537205192CD1A] #7, status:false, isInterrupted:false
2016-05-17 21:52:50,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:50,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-05-17 21:52:50,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:50,874 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:50,875 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-05-17 21:52:50,875 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-17 21:52:50,875 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,877 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:50,878 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 96CE1E94E5834D08A72B45BAC6B0CA4F
2016-05-17 21:52:50,879 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: D2EE6DA65982441D92E3469E7DAA39D0
2016-05-17 21:52:50,880 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] D2EE6DA65982441D92E3469E7DAA39D0
2016-05-17 21:52:50,882 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@5b1755a0; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@58f6aa18; comparator=cascading.tuple.hadoop.util.TupleComparator@2a193b49; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:50,882 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:50,882 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:50,882 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:50,891 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:50,891 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:50,891 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:50,892 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:50,892 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:50,893 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 96CE1E94E5834D08A72B45BAC6B0CA4F, for 1 inputs
2016-05-17 21:52:50,893 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:50,893 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:50,894 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FC155BF00B274480B4AC9C6A873ACAC0, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:50,895 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@52ffabb0; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@f60da62
2016-05-17 21:52:50,922 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,922 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC155BF00B274480B4AC9C6A873ACAC0, mem on close (mb), free: 87, total: 515, max: 672
2016-05-17 21:52:50,922 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D648923925F143B998A537205192CD1A
2016-05-17 21:52:50,923 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:50,923 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:50,923 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:50,924 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:50,924 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:50,924 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:50,924 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:50,925 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-05-17 21:52:50,925 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-05-17 21:52:50,937 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:50 Completed running task attempt: attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,942 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:50,943 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521970445_0001_1_01, eventCount=1
2016-05-17 21:52:50,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521970894, Process_Duration=28, Process_End_Time=1463521970922, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=6]]
2016-05-17 21:52:50,944 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:50,944 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:50,944 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, taskAttemptId=attempt_1463521970445_0001_1_01_000000_0, startTime=1463521970838, finishTime=1463521970943, timeTaken=105, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970894, Process_Duration=28, Process_End_Time=1463521970922, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=6
2016-05-17 21:52:50,944 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:50,944 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,944 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:50,944 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000002
2016-05-17 21:52:50,944 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521970445_0001_1_01_000000_0
2016-05-17 21:52:50,946 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:50,946 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_FINISHED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, taskId=task_1463521970445_0001_1_01_000000, startTime=1463521970838, finishTime=1463521970946, timeTaken=108, status=SUCCEEDED, successfulAttemptID=attempt_1463521970445_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970894, Process_Duration=28, Process_End_Time=1463521970922, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=6
2016-05-17 21:52:50,946 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:50,947 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] : 1
2016-05-17 21:52:50,947 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:50,948 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:50,948 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_FINISHED]: vertexName=FC155BF00B274480B4AC9C6A873ACAC0, vertexId=vertex_1463521970445_0001_1_01, initRequestedTime=1463521970666, initedTime=1463521970674, startRequestedTime=1463521970703, startedTime=1463521970703, finishTime=1463521970947, timeTaken=244, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540016640, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521970894, Process_Duration=28, Process_End_Time=1463521970922, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=6, vertexStats=firstTaskStartTime=1463521970838, firstTasksToStart=[ task_1463521970445_0001_1_01_000000 ], lastTaskFinishTime=1463521970943, lastTasksToFinish=[ task_1463521970445_0001_1_01_000000 ], minTaskDuration=105, maxTaskDuration=105, avgTaskDuration=105.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521970445_0001_1_01_000000 ], longestDurationTasks=[ task_1463521970445_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:50,948 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:50,949 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:50,949 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521970445_0001_00_000002, stoppedTime=1463521970949, exitStatus=0
2016-05-17 21:52:50,949 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:50,949 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] attempt: attempt_1463521970445_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:50,949 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:50,949 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:50,949 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:50,950 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:50,950 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521970445_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-17 21:52:50,951 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_STARTED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, taskId=task_1463521970445_0001_1_02_000000, scheduledTime=1463521970950, launchTime=1463521970950
2016-05-17 21:52:50,951 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:50,951 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:50,951 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:50,952 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:50,952 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: EF13735CD4F94A68AA6A6B96259AE027, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E521D1AA5A254346841DC9D7ED2FAA08, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,952 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:50,953 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:50,954 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521970445_0001_00_000003
2016-05-17 21:52:50,954 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:50,956 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521970445_0001_00_000003
2016-05-17 21:52:50,956 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-05-17 21:52:50,957 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:50,957 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521970445_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 21:52:50,958 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:50,958 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521970445_0001_00_000002, Exiting
2016-05-17 21:52:50,958 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521970445_0001_00_000002
2016-05-17 21:52:50,958 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521970445_0001_00_000002
2016-05-17 21:52:50,958 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521970445_0001_00_000003, launchTime=1463521970957
2016-05-17 21:52:50,958 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:50,960 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521970445_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:50,960 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521970445_0001_00_000002 completed successfully
2016-05-17 21:52:50,960 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:50,963 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000003
2016-05-17 21:52:50,965 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521970445_0001_1_02_000000_0] to container: [container_1463521970445_0001_00_000003]
2016-05-17 21:52:50,965 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:50,965 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521970445_0001_1_02_000000_0] started. Is using containerId: [container_1463521970445_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-17 21:52:50,965 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0, startTime=1463521970965, containerId=container_1463521970445_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521970445_0001_00_000003/teamcity, completedLogs=
2016-05-17 21:52:50,965 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:50,966 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:50,966 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:50,966 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521970445_0001_00_000003 given task: attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:50,966 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:50 Starting to run new task attempt: attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:50,967 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:50,967 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:50,967 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:50,967 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:50,967 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:50,967 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:50,968 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:50,968 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: EF13735CD4F94A68AA6A6B96259AE027, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E521D1AA5A254346841DC9D7ED2FAA08, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:50,968 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:50,968 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:50,968 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:50,969 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=FC155BF00B274480B4AC9C6A873ACAC0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:50,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:50,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,976 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:50,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E521D1AA5A254346841DC9D7ED2FAA08, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:50,973 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521970445_0001_1_02_000000_0 sent events: (0-1)
2016-05-17 21:52:50,978 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:50,978 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:50,980 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:50,980 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:50,980 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,980 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,981 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:50,981 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:50,982 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:50,986 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EF13735CD4F94A68AA6A6B96259AE027, ordinal: 2
2016-05-17 21:52:50,987 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EF13735CD4F94A68AA6A6B96259AE027, mem on start (mb), free: 81, total: 515, max: 672
2016-05-17 21:52:50,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:50,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:50,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:50,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:50,988 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:50,988 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:50,988 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:50,989 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:50,989 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:50,989 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, FC155BF00B274480B4AC9C6A873ACAC0, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:50,990 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E521D1AA5A254346841DC9D7ED2FAA08, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:50,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:50,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: FC155BF00B274480B4AC9C6A873ACAC0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:50,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:50,991 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:50,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,994 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:50,996 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:50,996 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: FC155BF00B274480B4AC9C6A873ACAC0 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:50,996 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:50,996 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:50,997 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:50,997 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:50,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:50,998 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:51,001 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-17 21:52:51,002 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: D2EE6DA65982441D92E3469E7DAA39D0
2016-05-17 21:52:51,002 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 37D0F5706771442CA9064C96EE708538
2016-05-17 21:52:51,002 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 37D0F5706771442CA9064C96EE708538
2016-05-17 21:52:51,003 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521970445_0001_1_01_000000_0_10030, runDuration: 92957, ]
2016-05-17 21:52:51,004 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@30a7803e; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@44d47f66; comparator=cascading.tuple.hadoop.util.TupleComparator@5af824d7; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:51,005 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:51,005 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:51,005 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:51,018 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521970445_0001_1_01_000000_0_10030] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1463521971018, TimeTaken=14, Rate=0.51 MB/s
2016-05-17 21:52:51,018 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:51,018 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-17 21:52:51,019 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [FC155BF00B274480B4AC9C6A873ACAC0] #8 in 15ms
2016-05-17 21:52:51,020 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [FC155BF00B274480B4AC9C6A873ACAC0] #8, status:false, isInterrupted:false
2016-05-17 21:52:51,027 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:51,028 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:51,028 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:51,029 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:51,029 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:51,029 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:51,029 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-05-17 21:52:51,030 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:51,030 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] D2EE6DA65982441D92E3469E7DAA39D0, for 1 inputs
2016-05-17 21:52:51,030 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:51,031 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-17 21:52:51,031 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:51,031 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EF13735CD4F94A68AA6A6B96259AE027, all 1 inputs ready in: 00:00:00.001
2016-05-17 21:52:51,031 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:51,032 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@4d95aab2; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@1cf9b3e9
2016-05-17 21:52:51,052 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:51,052 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EF13735CD4F94A68AA6A6B96259AE027, mem on close (mb), free: 199, total: 516, max: 672
2016-05-17 21:52:51,052 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: FC155BF00B274480B4AC9C6A873ACAC0
2016-05-17 21:52:51,052 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:51,053 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:51,053 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:51,053 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:51,053 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:51,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:51,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:51,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-05-17 21:52:51,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-05-17 21:52:51,060 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:51 Completed running task attempt: attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:51,065 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:51,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521970445_0001_1_02, eventCount=1
2016-05-17 21:52:51,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, taskAttemptId=attempt_1463521970445_0001_1_02_000000_0, startTime=1463521970965, finishTime=1463521971068, timeTaken=103, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=23, MERGE_PHASE_TIME=35, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971031, Process_Duration=21, Process_End_Time=1463521971052, Read_Duration=9, Tuples_Read=174, Tuples_Written=87, Write_Duration=5
2016-05-17 21:52:51,069 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:51,069 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521970445_0001_1_02_000000_0
2016-05-17 21:52:51,070 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_FINISHED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, taskId=task_1463521970445_0001_1_02_000000, startTime=1463521970965, finishTime=1463521971069, timeTaken=104, status=SUCCEEDED, successfulAttemptID=attempt_1463521970445_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=23, MERGE_PHASE_TIME=35, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971031, Process_Duration=21, Process_End_Time=1463521971052, Read_Duration=9, Tuples_Read=174, Tuples_Written=87, Write_Duration=5
2016-05-17 21:52:51,070 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:51,070 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] : 1
2016-05-17 21:52:51,070 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:51,073 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:51,073 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:51,073 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:51,075 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:51,075 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_FINISHED]: vertexName=EF13735CD4F94A68AA6A6B96259AE027, vertexId=vertex_1463521970445_0001_1_02, initRequestedTime=1463521970675, initedTime=1463521970680, startRequestedTime=1463521970707, startedTime=1463521970707, finishTime=1463521971071, timeTaken=364, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=23, MERGE_PHASE_TIME=35, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971031, Process_Duration=21, Process_End_Time=1463521971052, Read_Duration=9, Tuples_Read=174, Tuples_Written=87, Write_Duration=5, vertexStats=firstTaskStartTime=1463521970965, firstTasksToStart=[ task_1463521970445_0001_1_02_000000 ], lastTaskFinishTime=1463521971068, lastTasksToFinish=[ task_1463521970445_0001_1_02_000000 ], minTaskDuration=103, maxTaskDuration=103, avgTaskDuration=103.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521970445_0001_1_02_000000 ], longestDurationTasks=[ task_1463521970445_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:51,076 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:51,076 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=23, MERGE_PHASE_TIME=35, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521971031, Process_Duration=21, Process_End_Time=1463521971052, Read_Duration=9, Tuples_Read=174, Tuples_Written=87, Write_Duration=5]]
2016-05-17 21:52:51,077 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:51,077 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,077 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] attempt: attempt_1463521970445_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:51,077 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E521D1AA5A254346841DC9D7ED2FAA08
2016-05-17 21:52:51,078 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:51,079 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:51,078 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000003
2016-05-17 21:52:51,079 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:51,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521970445_0001_00_000003, stoppedTime=1463521971079, exitStatus=0
2016-05-17 21:52:51,080 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521970445_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-17 21:52:51,080 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:51,080 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:51,080 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521970445_0001_00_000003, Exiting
2016-05-17 21:52:51,081 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521970445_0001_00_000003
2016-05-17 21:52:51,081 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521970445_0001_00_000003
2016-05-17 21:52:51,081 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521970445_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:51,081 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521970445_0001_00_000003 completed successfully
2016-05-17 21:52:51,082 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:51,082 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_STARTED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, taskId=task_1463521970445_0001_1_03_000000, scheduledTime=1463521971082, launchTime=1463521971082
2016-05-17 21:52:51,082 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:51,083 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:51,083 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521970445_0001_00_000003
2016-05-17 21:52:51,083 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000003 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:51,083 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:51,084 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: E521D1AA5A254346841DC9D7ED2FAA08, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C61A8119CF93417FAAD84646D126A048, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:51,084 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:51,084 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:51,085 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521970445_0001_00_000004
2016-05-17 21:52:51,085 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:51,086 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521970445_0001_00_000004
2016-05-17 21:52:51,087 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:51,090 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521970445_0001_00_000004, launchTime=1463521971090
2016-05-17 21:52:51,090 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:51,090 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:51,090 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000004
2016-05-17 21:52:51,093 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521970445_0001_1_03_000000_0] to container: [container_1463521970445_0001_00_000004]
2016-05-17 21:52:51,093 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:51,093 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521970445_0001_00_000004 given task: attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,093 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,093 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521970445_0001_1_03_000000_0] started. Is using containerId: [container_1463521970445_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-17 21:52:51 Starting to run new task attempt: attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,094 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,094 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,094 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:51,095 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:51,095 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:51,095 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:51,095 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:51,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: E521D1AA5A254346841DC9D7ED2FAA08, VertexParallelism: 1, TaskAttemptID:attempt_1463521970445_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C61A8119CF93417FAAD84646D126A048, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:51,095 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:51,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0, startTime=1463521971093, containerId=container_1463521970445_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521970445_0001_00_000004/teamcity, completedLogs=
2016-05-17 21:52:51,096 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:51,097 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:51,095 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:51,098 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:51,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=EF13735CD4F94A68AA6A6B96259AE027, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:51,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C61A8119CF93417FAAD84646D126A048, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 21:52:51,099 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,104 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:51,104 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C61A8119CF93417FAAD84646D126A048
2016-05-17 21:52:51,104 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 21:52:51,104 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: C61A8119CF93417FAAD84646D126A048
2016-05-17 21:52:51,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:51,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,102 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521970445_0001_1_03_000000_0 sent events: (0-1)
2016-05-17 21:52:51,108 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:51,108 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:51,109 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,105 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:51,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,118 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 21:52:51,119 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C61A8119CF93417FAAD84646D126A048
2016-05-17 21:52:51,121 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E521D1AA5A254346841DC9D7ED2FAA08, ordinal: 3
2016-05-17 21:52:51,121 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E521D1AA5A254346841DC9D7ED2FAA08, mem on start (mb), free: 192, total: 516, max: 672
2016-05-17 21:52:51,121 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,121 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:51,121 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:51,122 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:51,122 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:51,123 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 21:52:51,123 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:51,123 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:51,125 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:51,125 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 21:52:51,125 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C61A8119CF93417FAAD84646D126A048, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 21:52:51,125 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, EF13735CD4F94A68AA6A6B96259AE027, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 21:52:51,125 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:51,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: EF13735CD4F94A68AA6A6B96259AE027 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:51,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:51,127 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:51,126 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:51,128 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:51,128 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: EF13735CD4F94A68AA6A6B96259AE027 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:51,128 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 21:52:51,128 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:51,129 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:51,130 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,134 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521970445_0001_1_02_000000_0_10033, runDuration: 85337, ]
2016-05-17 21:52:51,134 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-05-17 21:52:51,136 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521970445_0001_1_02_000000_0_10033] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1463521971136, TimeTaken=1, Rate=4.89 MB/s
2016-05-17 21:52:51,136 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,136 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 37D0F5706771442CA9064C96EE708538
2016-05-17 21:52:51,136 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:51,137 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: C61A8119CF93417FAAD84646D126A048
2016-05-17 21:52:51,137 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [EF13735CD4F94A68AA6A6B96259AE027] #9 in 3ms
2016-05-17 21:52:51,137 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-05-17 21:52:51,137 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [EF13735CD4F94A68AA6A6B96259AE027] #9, status:false, isInterrupted:false
2016-05-17 21:52:51,138 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 37D0F5706771442CA9064C96EE708538, for 1 inputs
2016-05-17 21:52:51,138 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:51,139 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-05-17 21:52:51,139 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:51,139 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:51,139 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-05-17 21:52:51,139 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,140 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:51,141 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E521D1AA5A254346841DC9D7ED2FAA08, all 1 inputs ready in: 00:00:00.003
2016-05-17 21:52:51,142 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@61ca2d91; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@2075bcbf
2016-05-17 21:52:51,159 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 21:52:51,160 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 21:52:51,161 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,161 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463521970445_0001_1_03_000000_0 given a go for committing the task output.
2016-05-17 21:52:51,162 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,163 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E521D1AA5A254346841DC9D7ED2FAA08, mem on close (mb), free: 189, total: 516, max: 672
2016-05-17 21:52:51,163 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: EF13735CD4F94A68AA6A6B96259AE027
2016-05-17 21:52:51,163 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:51,164 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:51,164 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:51,164 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:51,165 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:51 Completed running task attempt: attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,166 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:51,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521971141, Process_Duration=21, Process_End_Time=1463521971162, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-17 21:52:51,167 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:51,168 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,168 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, taskAttemptId=attempt_1463521970445_0001_1_03_000000_0, startTime=1463521971093, finishTime=1463521971167, timeTaken=74, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971141, Process_Duration=21, Process_End_Time=1463521971162, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:51,168 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521970445_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:51,168 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521970445_0001_1_03_000000_0
2016-05-17 21:52:51,169 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:TASK_FINISHED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, taskId=task_1463521970445_0001_1_03_000000, startTime=1463521971093, finishTime=1463521971168, timeTaken=75, status=SUCCEEDED, successfulAttemptID=attempt_1463521970445_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971141, Process_Duration=21, Process_End_Time=1463521971162, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:51,169 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:51,169 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521970445_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:51,169 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:51,170 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] : 1
2016-05-17 21:52:51,170 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:51,170 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:51,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:VERTEX_FINISHED]: vertexName=E521D1AA5A254346841DC9D7ED2FAA08, vertexId=vertex_1463521970445_0001_1_03, initRequestedTime=1463521970684, initedTime=1463521970696, startRequestedTime=1463521970709, startedTime=1463521970709, finishTime=1463521971170, timeTaken=461, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1936, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521971141, Process_Duration=21, Process_End_Time=1463521971162, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463521971093, firstTasksToStart=[ task_1463521970445_0001_1_03_000000 ], lastTaskFinishTime=1463521971167, lastTasksToFinish=[ task_1463521970445_0001_1_03_000000 ], minTaskDuration=74, maxTaskDuration=74, avgTaskDuration=74.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521970445_0001_1_03_000000 ], longestDurationTasks=[ task_1463521970445_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:51,171 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:51,172 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:51,171 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521970445_0001_00_000004
2016-05-17 21:52:51,171 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:51,172 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521970445_0001_00_000004, stoppedTime=1463521971172, exitStatus=0
2016-05-17 21:52:51,172 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:51,172 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:51,173 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463521970445_0001_1
2016-05-17 21:52:51,173 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463521970445_0001_1
2016-05-17 21:52:51,173 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:51,173 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521970445_0001_1_01 [FC155BF00B274480B4AC9C6A873ACAC0]
2016-05-17 21:52:51,174 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521970445_0001_1_00 [D648923925F143B998A537205192CD1A]
2016-05-17 21:52:51,174 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: C61A8119CF93417FAAD84646D126A048 for vertex: vertex_1463521970445_0001_1_03 [E521D1AA5A254346841DC9D7ED2FAA08]
2016-05-17 21:52:51,174 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521970445_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-17 21:52:51,175 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521970445_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:51,176 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521970445_0001_00_000004, Exiting
2016-05-17 21:52:51,176 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521970445_0001_00_000004
2016-05-17 21:52:51,176 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521970445_0001_00_000004
2016-05-17 21:52:51,177 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521970445_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:51,177 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521970445_0001_00_000004 completed successfully
2016-05-17 21:52:51,185 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521970445_0001_1_02 [EF13735CD4F94A68AA6A6B96259AE027]
2016-05-17 21:52:51,189 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463521970445_0001_1, queueSize=0
2016-05-17 21:52:51,189 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 21:52:51,190 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521970445_0001_1][Event:DAG_FINISHED]: dagId=dag_1463521970445_0001_1, startTime=1463521970663, finishTime=1463521971185, timeTaken=522, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15767, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=24, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2162163712, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=37, MERGE_PHASE_TIME=53, FIRST_EVENT_RECEIVED=17, LAST_EVENT_RECEIVED=17, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854087883857, Process_Duration=74, Process_End_Time=5854087883931, Read_Duration=23, Tuples_Read=303, Tuples_Written=317, Write_Duration=13, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-17 21:52:51,191 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463521970445_0001_1 finished with state: SUCCEEDED
2016-05-17 21:52:51,191 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521970445_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 21:52:51,191 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521970445_0001_00_000004
2016-05-17 21:52:51,192 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521970445_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:51 Completed Dag: dag_1463521970445_0001_1
2016-05-17 21:52:51,193 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463521970445_0001_1, dagState=SUCCEEDED
2016-05-17 21:52:51,195 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521970445_0001_1_post
2016-05-17 21:52:51,196 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,197 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 21:52:51,201 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463521970445_0001
2016-05-17 21:52:51,204 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 21:52:51,204 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 21:52:51,205 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 21:52:51,205 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 21:52:51,205 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 21:52:51,206 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 21:52:51,206 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 21:52:51,207 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 21:52:51,207 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 21:52:51,207 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 21:52:51,207 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 21:52:51,209 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 21:52:51,209 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 21:52:51,217 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C66309BD899D4147AE633B3A09F9DA4C/CFE682BE38764CA988DDCB7B185DD19B/.tez/application_1463521970445_0001
2016-05-17 21:52:51,218 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long/_temporary
2016-05-17 21:52:51,219 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:00.797
2016-05-17 21:52:51,304 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-05-17 21:52:51,340 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 21:52:51,341 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 21:52:51,399 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.029
2016-05-17 21:52:51,437 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.067
2016-05-17 21:52:51,437 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:51,438 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-17 21:52:51,438 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 21:52:51,457 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-05-17 21:52:51,487 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-05-17 21:52:51,635 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-17 21:52:51,636 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:51,637 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-05-17 21:52:51,637 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-17 21:52:51,637 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-17 21:52:51,638 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-17 21:52:51,639 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-05-17 21:52:51,639 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 21:52:51,652 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-17 21:52:51,657 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:51,658 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 21:52:51,658 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 21:52:51,667 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001 doesn't exist and is created
2016-05-17 21:52:51,684 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 21:52:51,684 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 21:52:51,684 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001
2016-05-17 21:52:51,688 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463521971657_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 21:52:51,700 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 21:52:51,701 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 21:52:51,703 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 21:52:51,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 21:52:51,704 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001/localmode-log-dir
2016-05-17 21:52:51,704 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001/localmode-log-dir/history.txt.appattempt_1463521971657_0001_000000, maxErrors=10
2016-05-17 21:52:51,705 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 21:52:51,705 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463521971657_0001_000000, appSubmitTime=1463521971687, launchTime=1463521971687
2016-05-17 21:52:51,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 21:52:51,718 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 21:52:51,720 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 21:52:51,722 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42974
2016-05-17 21:52:51,722 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 21:52:51,723 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 21:52:51,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463521971657_0001_000000, startTime=1463521971725
2016-05-17 21:52:51,725 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 21:52:51,725 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 21:52:51,784 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 21:52:51,784 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 21:52:51,784 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463521971657_0001, dagName=[9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-17 21:52:51,803 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-17 21:52:51,805 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463521971657_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001/localmode-log-dir/dag_1463521971657_0001_1.dot
2016-05-17 21:52:51,807 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001/localmode-log-dir/dag_1463521971657_0001_1-tez-dag.pb.txt
2016-05-17 21:52:51,850 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521971657_0001_1
2016-05-17 21:52:51,850 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,850 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-17 21:52:51 Running Dag: dag_1463521971657_0001_1
2016-05-17 21:52:51,864 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 21:52:51,865 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463521971657_0001_1, submitTime=1463521971803
2016-05-17 21:52:51,866 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:51,866 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 21:52:51,866 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 21:52:51,867 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463521971657_0001_1, initTime=1463521971865
2016-05-17 21:52:51,867 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521971657_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 21:52:51,868 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 21:52:51,868 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:DAG_STARTED]: dagID=dag_1463521971657_0001_1, startTime=1463521971868
2016-05-17 21:52:51,868 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 21:52:51,868 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463521971657_0001, dagName=[9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-17 21:52:51,868 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521971657_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 21:52:51,869 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: F00379BF4E994C37A406AA7B06832BF5 : {50B64CC122374447BD8E79B66588EB80={InputName=50B64CC122374447BD8E79B66588EB80}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 21:52:51,869 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 50B64CC122374447BD8E79B66588EB80, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 21:52:51,869 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,869 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,869 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,870 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]. Starting root input initializers: 1
2016-05-17 21:52:51,871 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 21:52:51,871 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:51,872 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 50B64CC122374447BD8E79B66588EB80 on vertex vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,874 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 21:52:51,875 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 50B64CC122374447BD8E79B66588EB80 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 21:52:51,877 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 21:52:51,877 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 21:52:51,880 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-05-17 21:52:51,881 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 50B64CC122374447BD8E79B66588EB80 on vertex vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,884 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:51,884 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:51,885 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:51,886 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:51,886 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, vertexId=vertex_1463521971657_0001_1_01, initRequestedTime=1463521971871, initedTime=1463521971886, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:51,888 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:51,888 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:51,890 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {50B64CC122374447BD8E79B66588EB80=forAllWorkUnits=true, update=[1]}
2016-05-17 21:52:51,891 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] parallelism set to 1
2016-05-17 21:52:51,891 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 21:52:51,891 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463521971657_0001, with dag id: dag_1463521971657_0001_1
2016-05-17 21:52:51,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521971657_0001_1_00, eventCount=1
2016-05-17 21:52:51,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, vertexId=vertex_1463521971657_0001_1_00, initRequestedTime=1463521971869, initedTime=1463521971894, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 21:52:51,894 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] with distanceFromRoot: 0
2016-05-17 21:52:51,894 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:51,895 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 21:52:51,896 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:51,901 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:51,902 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:51,902 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:51,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, vertexId=vertex_1463521971657_0001_1_02, initRequestedTime=1463521971896, initedTime=1463521971903, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:51,903 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:51,903 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:51,904 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:51,904 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,905 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521971657_0001_1_00, startRequestedTime=1463521971904, startedTime=1463521971904
2016-05-17 21:52:51,905 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:51,905 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:51,906 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,911 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 21:52:51,911 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,912 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,912 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,913 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=290703B91DD0433782057AA64132DB56, vertexId=vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 21:52:51,913 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=290703B91DD0433782057AA64132DB56, vertexId=vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,915 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=290703B91DD0433782057AA64132DB56, vertexName=C9D904C5B431400AB992E92794A1C9D4, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 21:52:51,916 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=290703B91DD0433782057AA64132DB56, vertexId=vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:51,917 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, vertexId=vertex_1463521971657_0001_1_03, initRequestedTime=1463521971906, initedTime=1463521971917, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 21:52:51,917 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 21:52:51,918 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] transitioned from NEW to INITED due to event V_INIT
2016-05-17 21:52:51,919 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:51,919 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_STARTED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, taskId=task_1463521971657_0001_1_00_000000, scheduledTime=1463521971918, launchTime=1463521971918
2016-05-17 21:52:51,919 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:51,920 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521971657_0001_1_00 for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] numStartedSources: 1 numSources: 1
2016-05-17 21:52:51,920 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] with distanceFromRoot: 1
2016-05-17 21:52:51,921 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:51,921 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: F00379BF4E994C37A406AA7B06832BF5 in vertex: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:51,922 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: EA5A8FBB1E534ACAA11E84EF54CC624E with 1 source tasks and 1 pending tasks
2016-05-17 21:52:51,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521971657_0001_1_01, startRequestedTime=1463521971921, startedTime=1463521971921
2016-05-17 21:52:51,922 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:51,922 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:51,923 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: F00379BF4E994C37A406AA7B06832BF5, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=50B64CC122374447BD8E79B66588EB80, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:51,924 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:51,924 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521971657_0001_1_01 for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] numStartedSources: 1 numSources: 1
2016-05-17 21:52:51,924 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] with distanceFromRoot: 2
2016-05-17 21:52:51,925 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:51,925 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: EA5A8FBB1E534ACAA11E84EF54CC624E in vertex: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:51,926 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A30CEB6E469442B5840165D4FF00FCB7 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:51,926 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:51,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521971657_0001_1_02, startRequestedTime=1463521971925, startedTime=1463521971925
2016-05-17 21:52:51,927 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:51,927 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463521971657_0001_1_02 for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] numStartedSources: 1 numSources: 1
2016-05-17 21:52:51,928 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] with distanceFromRoot: 3
2016-05-17 21:52:51,928 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A30CEB6E469442B5840165D4FF00FCB7 in vertex: C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:51,928 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 21:52:51,928 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C9D904C5B431400AB992E92794A1C9D4 with 1 source tasks and 1 pending tasks
2016-05-17 21:52:51,929 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 21:52:51,930 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463521971657_0001_1_03, startRequestedTime=1463521971928, startedTime=1463521971928
2016-05-17 21:52:51,930 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] transitioned from INITED to RUNNING due to event V_START
2016-05-17 21:52:51,930 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 21:52:51,931 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 21:52:51,931 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521971657_0001_00_000001
2016-05-17 21:52:51,932 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:51,935 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521971657_0001_00_000001
2016-05-17 21:52:51,935 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 21:52:51,936 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:51,937 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:51,937 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521971657_0001_00_000001, launchTime=1463521971937
2016-05-17 21:52:51,939 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000001
2016-05-17 21:52:51,940 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521971657_0001_1_00_000000_0] to container: [container_1463521971657_0001_00_000001]
2016-05-17 21:52:51,941 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:51,941 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521971657_0001_1_00_000000_0] started. Is using containerId: [container_1463521971657_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 21:52:51,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0, startTime=1463521971941, containerId=container_1463521971657_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521971657_0001_00_000001/teamcity, completedLogs=
2016-05-17 21:52:51,942 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:51,942 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:51,942 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:51,942 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521971657_0001_00_000001 given task: attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51,943 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51 Starting to run new task attempt: attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51,943 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51,943 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:51,944 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:51,944 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:51,944 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:51,944 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:51,944 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:51,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: F00379BF4E994C37A406AA7B06832BF5, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=50B64CC122374447BD8E79B66588EB80, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:51,945 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:51,945 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51,945 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:51,945 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,946 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=50B64CC122374447BD8E79B66588EB80, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 21:52:51,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:51,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,948 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,951 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,956 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521971657_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 21:52:51,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:51,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:51,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:51,961 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:51,961 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:51,962 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:51,963 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:51,963 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 21:52:51,964 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,969 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F00379BF4E994C37A406AA7B06832BF5, ordinal: 0
2016-05-17 21:52:51,969 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F00379BF4E994C37A406AA7B06832BF5, mem on start (mb), free: 59, total: 516, max: 672
2016-05-17 21:52:51,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:51,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:51,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:51,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:51,970 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:51,971 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 21:52:51,971 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:51,971 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:51,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:51,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 21:52:51,972 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 50B64CC122374447BD8E79B66588EB80, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 21:52:51,972 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, EA5A8FBB1E534ACAA11E84EF54CC624E, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 21:52:51,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:51,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 50B64CC122374447BD8E79B66588EB80 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:51,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:51,974 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:51,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,974 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:51,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:51,975 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:51,978 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 21:52:51,979 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@5de41cab
2016-05-17 21:52:51,979 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:51,980 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 50B64CC122374447BD8E79B66588EB80
2016-05-17 21:52:51,981 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 5CB9CB434EE149BAB3F4E6E5D57467C5
2016-05-17 21:52:51,981 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 5CB9CB434EE149BAB3F4E6E5D57467C5
2016-05-17 21:52:51,983 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@5de41cab
2016-05-17 21:52:51,983 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 21:52:51,984 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 21:52:51,984 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@68e7e099; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@4f4a929d; comparator=cascading.tuple.hadoop.util.TupleComparator@28dfaa16; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:51,985 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:51,985 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:51,985 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,010 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 21:52:52,010 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 21:52:52,010 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 21:52:52,010 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 21:52:52,011 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 21:52:52,011 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-05-17 21:52:52,011 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 21:52:52,012 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F00379BF4E994C37A406AA7B06832BF5, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:52,015 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:52,015 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F00379BF4E994C37A406AA7B06832BF5, mem on close (mb), free: 161, total: 517, max: 672
2016-05-17 21:52:52,016 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:52,016 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:52,016 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-05-17 21:52:52,016 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-05-17 21:52:52,019 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:52 Completed running task attempt: attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:52,023 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:52,024 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521971657_0001_1_00, eventCount=1
2016-05-17 21:52:52,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463521972012, Process_Duration=3, Process_End_Time=1463521972015, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-17 21:52:52,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, taskAttemptId=attempt_1463521971657_0001_1_00_000000_0, startTime=1463521971941, finishTime=1463521972025, timeTaken=84, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972012, Process_Duration=3, Process_End_Time=1463521972015, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:52,025 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:52,025 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:52,026 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,025 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:52,026 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000001
2016-05-17 21:52:52,026 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521971657_0001_1_00_000000_0
2016-05-17 21:52:52,026 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:52,027 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:52,027 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_FINISHED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, taskId=task_1463521971657_0001_1_00_000000, startTime=1463521971941, finishTime=1463521972026, timeTaken=85, status=SUCCEEDED, successfulAttemptID=attempt_1463521971657_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972012, Process_Duration=3, Process_End_Time=1463521972015, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-05-17 21:52:52,027 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:52,028 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] : 1
2016-05-17 21:52:52,028 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:52,030 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:52,030 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_FINISHED]: vertexName=F00379BF4E994C37A406AA7B06832BF5, vertexId=vertex_1463521971657_0001_1_00, initRequestedTime=1463521971869, initedTime=1463521971894, startRequestedTime=1463521971904, startedTime=1463521971904, finishTime=1463521972028, timeTaken=124, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=11, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972012, Process_Duration=3, Process_End_Time=1463521972015, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463521971941, firstTasksToStart=[ task_1463521971657_0001_1_00_000000 ], lastTaskFinishTime=1463521972025, lastTasksToFinish=[ task_1463521971657_0001_1_00_000000 ], minTaskDuration=84, maxTaskDuration=84, avgTaskDuration=84.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521971657_0001_1_00_000000 ], longestDurationTasks=[ task_1463521971657_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:52,030 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:52,031 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:52,031 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521971657_0001_00_000001, stoppedTime=1463521972031, exitStatus=0
2016-05-17 21:52:52,031 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:52,031 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] attempt: attempt_1463521971657_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:52,031 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,032 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:52,032 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:52,032 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:52,032 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521971657_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 21:52:52,033 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_STARTED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, taskId=task_1463521971657_0001_1_01_000000, scheduledTime=1463521972033, launchTime=1463521972033
2016-05-17 21:52:52,033 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:52,033 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:52,033 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:52,034 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:52,034 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: EA5A8FBB1E534ACAA11E84EF54CC624E, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F00379BF4E994C37A406AA7B06832BF5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:52,034 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:52,034 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:52,036 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521971657_0001_00_000002
2016-05-17 21:52:52,036 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:52,037 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521971657_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 21:52:52,037 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:52,037 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521971657_0001_00_000001, Exiting
2016-05-17 21:52:52,037 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521971657_0001_00_000001
2016-05-17 21:52:52,037 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521971657_0001_00_000001
2016-05-17 21:52:52,040 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521971657_0001_00_000002
2016-05-17 21:52:52,040 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521971657_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:52,040 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521971657_0001_00_000001 completed successfully
2016-05-17 21:52:52,041 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:52,041 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:52,043 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:52,043 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521971657_0001_00_000002, launchTime=1463521972043
2016-05-17 21:52:52,043 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:52,044 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000002
2016-05-17 21:52:52,045 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521971657_0001_1_01_000000_0] to container: [container_1463521971657_0001_00_000002]
2016-05-17 21:52:52,045 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:52,045 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521971657_0001_1_01_000000_0] started. Is using containerId: [container_1463521971657_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 21:52:52,046 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0, startTime=1463521972045, containerId=container_1463521971657_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521971657_0001_00_000002/teamcity, completedLogs=
2016-05-17 21:52:52,046 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:52,046 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:52,046 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:52,046 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521971657_0001_00_000002 given task: attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,047 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52 Starting to run new task attempt: attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,047 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,048 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,048 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:52,048 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:52,049 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:52,049 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:52,049 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:52,049 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: EA5A8FBB1E534ACAA11E84EF54CC624E, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F00379BF4E994C37A406AA7B06832BF5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:52,050 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:52,050 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,050 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:52,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,051 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521971657_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 21:52:52,052 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,052 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:52,052 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=F00379BF4E994C37A406AA7B06832BF5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:52,053 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:52,053 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,053 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:52,054 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,055 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:52,055 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,056 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,056 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:52,056 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:52,056 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:52,057 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,057 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,066 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EA5A8FBB1E534ACAA11E84EF54CC624E, ordinal: 1
2016-05-17 21:52:52,066 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA5A8FBB1E534ACAA11E84EF54CC624E, mem on start (mb), free: 155, total: 517, max: 672
2016-05-17 21:52:52,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:52,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:52,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:52,066 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:52,067 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:52,067 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:52,067 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:52,067 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:52,068 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A30CEB6E469442B5840165D4FF00FCB7, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:52,068 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, F00379BF4E994C37A406AA7B06832BF5, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:52,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:52,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: F00379BF4E994C37A406AA7B06832BF5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:52,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:52,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:52,070 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,071 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:52,071 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: F00379BF4E994C37A406AA7B06832BF5 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:52,071 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:52,071 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:52,072 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:52,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,073 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:52,073 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,074 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521971657_0001_1_00_000000_0_10039, runDuration: 64718, ]
2016-05-17 21:52:52,075 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521971657_0001_1_00_000000_0_10039] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1463521972075, TimeTaken=0, Rate=0.00 MB/s
2016-05-17 21:52:52,076 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,076 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:52,078 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-05-17 21:52:52,079 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [F00379BF4E994C37A406AA7B06832BF5] #10 in 4ms
2016-05-17 21:52:52,080 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [F00379BF4E994C37A406AA7B06832BF5] #10, status:false, isInterrupted:false
2016-05-17 21:52:52,080 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 5CB9CB434EE149BAB3F4E6E5D57467C5
2016-05-17 21:52:52,080 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: C290AD41BC7644B9970C190775C17741
2016-05-17 21:52:52,080 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:52,081 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C290AD41BC7644B9970C190775C17741
2016-05-17 21:52:52,081 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-05-17 21:52:52,081 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,082 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:52,082 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-17 21:52:52,082 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,083 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:52,084 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@1cccea68; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@5131609d; comparator=cascading.tuple.hadoop.util.TupleComparator@be0d10; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:52,084 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:52,084 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:52,084 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,093 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:52,093 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:52,093 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:52,093 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:52,094 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:52,095 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 5CB9CB434EE149BAB3F4E6E5D57467C5, for 1 inputs
2016-05-17 21:52:52,096 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:52,096 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 21:52:52,097 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EA5A8FBB1E534ACAA11E84EF54CC624E, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:52,098 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@7ad78ab8; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@5cd47509
2016-05-17 21:52:52,125 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,125 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA5A8FBB1E534ACAA11E84EF54CC624E, mem on close (mb), free: 88, total: 517, max: 672
2016-05-17 21:52:52,126 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: F00379BF4E994C37A406AA7B06832BF5
2016-05-17 21:52:52,126 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:52,127 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:52,127 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,127 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:52,128 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:52,128 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:52,128 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:52,129 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-05-17 21:52:52,129 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-05-17 21:52:52,136 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:52 Completed running task attempt: attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,140 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:52,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521971657_0001_1_01, eventCount=1
2016-05-17 21:52:52,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, taskAttemptId=attempt_1463521971657_0001_1_01_000000_0, startTime=1463521972045, finishTime=1463521972141, timeTaken=96, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972097, Process_Duration=28, Process_End_Time=1463521972125, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=8
2016-05-17 21:52:52,142 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:52,142 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521971657_0001_1_01_000000_0
2016-05-17 21:52:52,142 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:52,143 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_FINISHED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, taskId=task_1463521971657_0001_1_01_000000, startTime=1463521972045, finishTime=1463521972142, timeTaken=97, status=SUCCEEDED, successfulAttemptID=attempt_1463521971657_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972097, Process_Duration=28, Process_End_Time=1463521972125, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=8
2016-05-17 21:52:52,143 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:52,143 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:52,143 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] : 1
2016-05-17 21:52:52,143 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:52,144 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:52,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:52,145 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_FINISHED]: vertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, vertexId=vertex_1463521971657_0001_1_01, initRequestedTime=1463521971871, initedTime=1463521971886, startRequestedTime=1463521971921, startedTime=1463521971921, finishTime=1463521972144, timeTaken=223, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972097, Process_Duration=28, Process_End_Time=1463521972125, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=8, vertexStats=firstTaskStartTime=1463521972045, firstTasksToStart=[ task_1463521971657_0001_1_01_000000 ], lastTaskFinishTime=1463521972141, lastTasksToFinish=[ task_1463521971657_0001_1_01_000000 ], minTaskDuration=96, maxTaskDuration=96, avgTaskDuration=96.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521971657_0001_1_01_000000 ], longestDurationTasks=[ task_1463521971657_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:52,145 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:52,145 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] attempt: attempt_1463521971657_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:52,145 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,146 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:52,146 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521972097, Process_Duration=28, Process_End_Time=1463521972125, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=8]]
2016-05-17 21:52:52,146 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:52,146 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:52,147 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,147 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:52,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_STARTED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, taskId=task_1463521971657_0001_1_02_000000, scheduledTime=1463521972147, launchTime=1463521972147
2016-05-17 21:52:52,147 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:52,148 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000002
2016-05-17 21:52:52,148 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:52,148 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:52,148 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521971657_0001_00_000002, stoppedTime=1463521972148, exitStatus=0
2016-05-17 21:52:52,149 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:52,149 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:52,149 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463521971657_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-17 21:52:52,149 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A30CEB6E469442B5840165D4FF00FCB7, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C9D904C5B431400AB992E92794A1C9D4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:52,149 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:52,150 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 21:52:52,150 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:52,150 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521971657_0001_00_000003
2016-05-17 21:52:52,151 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:52,152 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521971657_0001_00_000003
2016-05-17 21:52:52,152 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-05-17 21:52:52,153 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:52,153 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:52,153 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521971657_0001_00_000003, launchTime=1463521972153
2016-05-17 21:52:52,154 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000003
2016-05-17 21:52:52,155 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521971657_0001_1_02_000000_0] to container: [container_1463521971657_0001_00_000003]
2016-05-17 21:52:52,155 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:52,155 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521971657_0001_1_02_000000_0] started. Is using containerId: [container_1463521971657_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-17 21:52:52,156 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0, startTime=1463521972155, containerId=container_1463521971657_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521971657_0001_00_000003/teamcity, completedLogs=
2016-05-17 21:52:52,156 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:52,156 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:52,156 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:52,157 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521971657_0001_00_000003 given task: attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,157 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52 Starting to run new task attempt: attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,158 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,158 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,158 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:52,158 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:52,159 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:52,159 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:52,159 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:52,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A30CEB6E469442B5840165D4FF00FCB7, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C9D904C5B431400AB992E92794A1C9D4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 21:52:52,160 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:52,160 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521971657_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 21:52:52,160 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:52,160 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521971657_0001_00_000002, Exiting
2016-05-17 21:52:52,160 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521971657_0001_00_000002
2016-05-17 21:52:52,161 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521971657_0001_00_000002
2016-05-17 21:52:52,161 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521971657_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:52,161 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521971657_0001_00_000002 completed successfully
2016-05-17 21:52:52,162 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 21:52:52,162 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,162 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:52,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,164 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,165 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521971657_0001_1_02_000000_0 sent events: (0-1)
2016-05-17 21:52:52,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=EA5A8FBB1E534ACAA11E84EF54CC624E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:52,166 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:52,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C9D904C5B431400AB992E92794A1C9D4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 21:52:52,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:52,168 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:52,168 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:52,168 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:52,169 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:52,171 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 21:52:52,171 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:52,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:52,177 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A30CEB6E469442B5840165D4FF00FCB7, ordinal: 2
2016-05-17 21:52:52,178 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A30CEB6E469442B5840165D4FF00FCB7, mem on start (mb), free: 83, total: 517, max: 672
2016-05-17 21:52:52,178 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,178 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:52,178 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:52,179 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:52,179 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:52,179 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:52,180 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 21:52:52,180 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:52,180 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-05-17 21:52:52,180 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, EA5A8FBB1E534ACAA11E84EF54CC624E, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-05-17 21:52:52,181 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C9D904C5B431400AB992E92794A1C9D4, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-05-17 21:52:52,181 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:52,181 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: EA5A8FBB1E534ACAA11E84EF54CC624E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:52,182 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:52,182 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:52,181 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,183 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,184 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:52,185 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: EA5A8FBB1E534ACAA11E84EF54CC624E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:52,185 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-05-17 21:52:52,185 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:52,185 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:52,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:52,188 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,188 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521971657_0001_1_01_000000_0_10041, runDuration: 84877, ]
2016-05-17 21:52:52,193 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-17 21:52:52,194 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: C290AD41BC7644B9970C190775C17741
2016-05-17 21:52:52,194 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: F2105A74F75149029AA9C3BECDA7BA8F
2016-05-17 21:52:52,195 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] F2105A74F75149029AA9C3BECDA7BA8F
2016-05-17 21:52:52,194 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521971657_0001_1_01_000000_0_10041] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1463521972194, TimeTaken=1, Rate=7.17 MB/s
2016-05-17 21:52:52,196 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,196 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-17 21:52:52,197 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@75f250ac; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@778e88b2; comparator=cascading.tuple.hadoop.util.TupleComparator@2eb09b70; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 21:52:52,197 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 21:52:52,197 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [EA5A8FBB1E534ACAA11E84EF54CC624E] #11 in 4ms
2016-05-17 21:52:52,198 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [EA5A8FBB1E534ACAA11E84EF54CC624E] #11, status:false, isInterrupted:false
2016-05-17 21:52:52,198 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 21:52:52,198 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,207 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:52,208 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-05-17 21:52:52,208 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,208 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:52,209 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-17 21:52:52,209 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,209 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:52,216 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-05-17 21:52:52,217 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-05-17 21:52:52,217 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-05-17 21:52:52,217 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-05-17 21:52:52,218 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-05-17 21:52:52,219 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C290AD41BC7644B9970C190775C17741, for 1 inputs
2016-05-17 21:52:52,219 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A30CEB6E469442B5840165D4FF00FCB7, all 1 inputs ready in: 00:00:00.000
2016-05-17 21:52:52,221 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@1cf0850b; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@72aea404
2016-05-17 21:52:52,241 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,241 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A30CEB6E469442B5840165D4FF00FCB7, mem on close (mb), free: 184, total: 502, max: 672
2016-05-17 21:52:52,241 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: EA5A8FBB1E534ACAA11E84EF54CC624E
2016-05-17 21:52:52,241 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:52,242 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:52,242 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,243 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:52,243 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:52,243 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 21:52:52,244 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 21:52:52,244 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-05-17 21:52:52,244 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-05-17 21:52:52,248 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 21:52:52 Completed running task attempt: attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,253 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:52,254 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463521971657_0001_1_02, eventCount=1
2016-05-17 21:52:52,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521972219, Process_Duration=21, Process_End_Time=1463521972240, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4]]
2016-05-17 21:52:52,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, taskAttemptId=attempt_1463521971657_0001_1_02_000000_0, startTime=1463521972155, finishTime=1463521972254, timeTaken=99, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972219, Process_Duration=21, Process_End_Time=1463521972240, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-05-17 21:52:52,255 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:52,255 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:52,254 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:52,255 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521971657_0001_1_02_000000_0
2016-05-17 21:52:52,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_FINISHED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, taskId=task_1463521971657_0001_1_02_000000, startTime=1463521972155, finishTime=1463521972257, timeTaken=102, status=SUCCEEDED, successfulAttemptID=attempt_1463521971657_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972219, Process_Duration=21, Process_End_Time=1463521972240, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-05-17 21:52:52,258 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:52,258 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] : 1
2016-05-17 21:52:52,258 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:52,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_FINISHED]: vertexName=A30CEB6E469442B5840165D4FF00FCB7, vertexId=vertex_1463521971657_0001_1_02, initRequestedTime=1463521971896, initedTime=1463521971903, startRequestedTime=1463521971925, startedTime=1463521971925, finishTime=1463521972258, timeTaken=333, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=8, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972219, Process_Duration=21, Process_End_Time=1463521972240, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4, vertexStats=firstTaskStartTime=1463521972155, firstTasksToStart=[ task_1463521971657_0001_1_02_000000 ], lastTaskFinishTime=1463521972254, lastTasksToFinish=[ task_1463521971657_0001_1_02_000000 ], minTaskDuration=99, maxTaskDuration=99, avgTaskDuration=99.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521971657_0001_1_02_000000 ], longestDurationTasks=[ task_1463521971657_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:52,259 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:52,259 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:52,260 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521971657_0001_00_000003, stoppedTime=1463521972260, exitStatus=0
2016-05-17 21:52:52,260 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:52,260 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] attempt: attempt_1463521971657_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 21:52:52,260 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C9D904C5B431400AB992E92794A1C9D4
2016-05-17 21:52:52,260 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:52,260 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:52,261 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:52,262 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:52,262 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,262 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000003
2016-05-17 21:52:52,262 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:52,262 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521971657_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-17 21:52:52,262 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:52,262 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521971657_0001_00_000003, Exiting
2016-05-17 21:52:52,262 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521971657_0001_00_000003
2016-05-17 21:52:52,263 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521971657_0001_00_000003
2016-05-17 21:52:52,263 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521971657_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:52,263 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521971657_0001_00_000003 completed successfully
2016-05-17 21:52:52,263 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:52,263 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521971657_0001_00_000003
2016-05-17 21:52:52,263 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_STARTED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, taskId=task_1463521971657_0001_1_03_000000, scheduledTime=1463521972263, launchTime=1463521972263
2016-05-17 21:52:52,263 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 21:52:52,264 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 21:52:52,266 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000003 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:52,266 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:52,267 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C9D904C5B431400AB992E92794A1C9D4, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=290703B91DD0433782057AA64132DB56, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:52,267 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 21:52:52,267 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 21:52:52,269 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463521971657_0001_00_000004
2016-05-17 21:52:52,269 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 21:52:52,270 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463521971657_0001_00_000004
2016-05-17 21:52:52,271 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 21:52:52,273 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 21:52:52,273 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 21:52:52,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463521971657_0001_00_000004, launchTime=1463521972272
2016-05-17 21:52:52,274 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000004
2016-05-17 21:52:52,274 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463521971657_0001_1_03_000000_0] to container: [container_1463521971657_0001_00_000004]
2016-05-17 21:52:52,275 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 21:52:52,275 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463521971657_0001_00_000004 given task: attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,275 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52 Starting to run new task attempt: attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,276 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,276 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,276 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 21:52:52,276 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 21:52:52,276 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 21:52:52,276 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 21:52:52,276 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 21:52:52,276 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: C9D904C5B431400AB992E92794A1C9D4, VertexParallelism: 1, TaskAttemptID:attempt_1463521971657_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=290703B91DD0433782057AA64132DB56, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 21:52:52,276 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 21:52:52,275 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463521971657_0001_1_03_000000_0] started. Is using containerId: [container_1463521971657_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-17 21:52:52,277 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,277 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 21:52:52,277 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463521971657_0001_1_03_000000_0 sent events: (0-1)
2016-05-17 21:52:52,277 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,277 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0, startTime=1463521972275, containerId=container_1463521971657_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463521971657_0001_00_000004/teamcity, completedLogs=
2016-05-17 21:52:52,279 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 21:52:52,279 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 21:52:52,277 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 21:52:52,280 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A30CEB6E469442B5840165D4FF00FCB7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 21:52:52,280 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=290703B91DD0433782057AA64132DB56, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 21:52:52,280 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 21:52:52,280 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,281 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,282 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 21:52:52,283 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 290703B91DD0433782057AA64132DB56
2016-05-17 21:52:52,283 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 21:52:52,283 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 290703B91DD0433782057AA64132DB56
2016-05-17 21:52:52,283 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 21:52:52,283 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 21:52:52,283 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 21:52:52,283 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,284 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,293 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C9D904C5B431400AB992E92794A1C9D4, ordinal: 3
2016-05-17 21:52:52,293 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9D904C5B431400AB992E92794A1C9D4, mem on start (mb), free: 178, total: 502, max: 672
2016-05-17 21:52:52,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 21:52:52,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 21:52:52,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 21:52:52,299 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 21:52:52,299 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 290703B91DD0433782057AA64132DB56
2016-05-17 21:52:52,299 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 21:52:52,299 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 21:52:52,300 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 21:52:52,300 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 21:52:52,300 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 21:52:52,300 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 21:52:52,300 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 21:52:52,300 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 290703B91DD0433782057AA64132DB56, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 21:52:52,300 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A30CEB6E469442B5840165D4FF00FCB7, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 21:52:52,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 21:52:52,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A30CEB6E469442B5840165D4FF00FCB7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 21:52:52,301 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,301 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 21:52:52,301 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 21:52:52,301 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 21:52:52,302 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 21:52:52,302 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A30CEB6E469442B5840165D4FF00FCB7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 21:52:52,302 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 21:52:52,303 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 21:52:52,305 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 21:52:52,305 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,306 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 21:52:52,306 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,307 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463521971657_0001_1_02_000000_0_10045, runDuration: 81889, ]
2016-05-17 21:52:52,312 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463521971657_0001_1_02_000000_0_10045] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1463521972312, TimeTaken=1, Rate=4.89 MB/s
2016-05-17 21:52:52,312 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,312 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 21:52:52,312 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [A30CEB6E469442B5840165D4FF00FCB7] #12 in 1ms
2016-05-17 21:52:52,312 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-05-17 21:52:52,312 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A30CEB6E469442B5840165D4FF00FCB7] #12, status:false, isInterrupted:false
2016-05-17 21:52:52,314 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: F2105A74F75149029AA9C3BECDA7BA8F
2016-05-17 21:52:52,314 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 21:52:52,314 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: 290703B91DD0433782057AA64132DB56
2016-05-17 21:52:52,314 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-05-17 21:52:52,314 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] F2105A74F75149029AA9C3BECDA7BA8F, for 1 inputs
2016-05-17 21:52:52,316 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-05-17 21:52:52,316 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,316 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 21:52:52,317 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-05-17 21:52:52,317 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C9D904C5B431400AB992E92794A1C9D4, all 1 inputs ready in: 00:00:00.003
2016-05-17 21:52:52,318 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,319 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@4d819c30; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@78353b10
2016-05-17 21:52:52,322 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 21:52:52,330 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 21:52:52,330 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 21:52:52,331 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,332 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463521971657_0001_1_03_000000_0 given a go for committing the task output.
2016-05-17 21:52:52,333 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,333 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9D904C5B431400AB992E92794A1C9D4, mem on close (mb), free: 174, total: 502, max: 672
2016-05-17 21:52:52,333 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A30CEB6E469442B5840165D4FF00FCB7
2016-05-17 21:52:52,334 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 21:52:52,334 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 21:52:52,335 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 21:52:52,335 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 21:52:52,335 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 21:52:52 Completed running task attempt: attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,336 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0, fatalErrorOccurred=false
2016-05-17 21:52:52,338 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463521972317, Process_Duration=15, Process_End_Time=1463521972332, Read_Duration=8, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-17 21:52:52,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, taskAttemptId=attempt_1463521971657_0001_1_03_000000_0, startTime=1463521972275, finishTime=1463521972337, timeTaken=62, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972317, Process_Duration=15, Process_End_Time=1463521972332, Read_Duration=8, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:52,338 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463521971657_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 21:52:52,338 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 21:52:52,338 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 21:52:52,339 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 21:52:52,338 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463521971657_0001_1_03_000000_0
2016-05-17 21:52:52,340 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:TASK_FINISHED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, taskId=task_1463521971657_0001_1_03_000000, startTime=1463521972275, finishTime=1463521972339, timeTaken=64, status=SUCCEEDED, successfulAttemptID=attempt_1463521971657_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972317, Process_Duration=15, Process_End_Time=1463521972332, Read_Duration=8, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-17 21:52:52,340 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 21:52:52,340 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463521971657_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 21:52:52,340 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] : 1
2016-05-17 21:52:52,340 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:52,341 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:VERTEX_FINISHED]: vertexName=C9D904C5B431400AB992E92794A1C9D4, vertexId=vertex_1463521971657_0001_1_03, initRequestedTime=1463521971906, initedTime=1463521971917, startRequestedTime=1463521971928, startedTime=1463521971928, finishTime=1463521972340, timeTaken=412, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=526385152, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463521972317, Process_Duration=15, Process_End_Time=1463521972332, Read_Duration=8, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463521972275, firstTasksToStart=[ task_1463521971657_0001_1_03_000000 ], lastTaskFinishTime=1463521972337, lastTasksToFinish=[ task_1463521971657_0001_1_03_000000 ], minTaskDuration=62, maxTaskDuration=62, avgTaskDuration=62.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463521971657_0001_1_03_000000 ], longestDurationTasks=[ task_1463521971657_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 21:52:52,341 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 21:52:52,341 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 21:52:52,342 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463521971657_0001_00_000004
2016-05-17 21:52:52,341 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 21:52:52,343 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463521971657_0001_00_000004, stoppedTime=1463521972343, exitStatus=0
2016-05-17 21:52:52,344 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 21:52:52,344 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 21:52:52,344 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-17 21:52:52,344 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463521971657_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-17 21:52:52,344 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463521971657_0001_1
2016-05-17 21:52:52,344 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463521971657_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 21:52:52,344 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463521971657_0001_1
2016-05-17 21:52:52,344 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463521971657_0001_00_000004, Exiting
2016-05-17 21:52:52,345 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463521971657_0001_00_000004
2016-05-17 21:52:52,345 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463521971657_0001_00_000004
2016-05-17 21:52:52,345 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 290703B91DD0433782057AA64132DB56 for vertex: vertex_1463521971657_0001_1_03 [C9D904C5B431400AB992E92794A1C9D4]
2016-05-17 21:52:52,347 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463521971657_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 21:52:52,347 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463521971657_0001_00_000004 completed successfully
2016-05-17 21:52:52,355 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521971657_0001_1_01 [EA5A8FBB1E534ACAA11E84EF54CC624E]
2016-05-17 21:52:52,356 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521971657_0001_1_02 [A30CEB6E469442B5840165D4FF00FCB7]
2016-05-17 21:52:52,356 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463521971657_0001_1_00 [F00379BF4E994C37A406AA7B06832BF5]
2016-05-17 21:52:52,359 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463521971657_0001_1, queueSize=0
2016-05-17 21:52:52,360 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 21:52:52,360 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463521971657_0001_1][Event:DAG_FINISHED]: dagId=dag_1463521971657_0001_1, startTime=1463521971868, finishTime=1463521972356, timeTaken=488, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=19, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2136997888, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=33, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=13, LAST_EVENT_RECEIVED=13, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854087888645, Process_Duration=67, Process_End_Time=5854087888712, Read_Duration=14, Tuples_Read=303, Tuples_Written=317, Write_Duration=13, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-17 21:52:52,360 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463521971657_0001_1 finished with state: SUCCEEDED
2016-05-17 21:52:52,360 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463521971657_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 21:52:52,361 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463521971657_0001_00_000004
2016-05-17 21:52:52,361 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463521971657_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 21:52:52 Completed Dag: dag_1463521971657_0001_1
2016-05-17 21:52:52,362 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463521971657_0001_1, dagState=SUCCEEDED
2016-05-17 21:52:52,362 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463521971657_0001_1_post
2016-05-17 21:52:52,362 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 21:52:52,363 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 21:52:52,369 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463521971657_0001
2016-05-17 21:52:52,372 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 21:52:52,372 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 21:52:52,372 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 21:52:52,372 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 21:52:52,372 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 21:52:52,373 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 21:52:52,373 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 21:52:52,374 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 21:52:52,374 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 21:52:52,374 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 21:52:52,374 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 21:52:52,375 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 21:52:52,375 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 21:52:52,382 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/9624AFB821574AA99E8EB5893452E492/E6B4FC21089B46C5ADFF117774D32F79/.tez/application_1463521971657_0001
2016-05-17 21:52:52,383 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short/_temporary
2016-05-17 21:52:52,383 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:00.747
2016-05-17 21:52:52,465 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short