2016-01-14 16:29:56,731 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(195)) - found 1 test platforms from classpath
2016-01-14 16:29:56,765 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(244)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-01-14 16:29:56,801 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-01-14 16:29:57,185 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-14 16:29:57,549 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(73)) - platform property overrides:
2016-01-14 16:29:57,805 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:29:57,806 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:29:57,919 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: E3C4D7DE265A4AC28A787666BB0CEA75
2016-01-14 16:29:58,199 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.153
2016-01-14 16:29:58,281 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.236
2016-01-14 16:29:58,282 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:29:58,283 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:29:58,284 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:29:58,425 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:29:58,632 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 16:29:58,906 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.0.4
2016-01-14 16:29:58,908 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:29:58,908 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:29:58,909 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 16:29:58,909 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 16:29:58,909 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 16:29:58,910 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 16:29:58,911 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-01-14 16:29:58,911 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:29:58,938 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:29:59,134 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:29:59,140 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:29:59,140 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 16:29:59,182 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001 doesn't exist and is created
2016-01-14 16:29:59,240 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:29:59,241 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,242 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001
2016-01-14 16:29:59,270 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452788999139_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:29:59,310 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:29:59,324 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:29:59,341 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-01-14 16:29:59,341 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-01-14 16:29:59,341 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,348 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:29:59,352 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:29:59,355 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir
2016-01-14 16:29:59,355 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/history.txt.appattempt_1452788999139_0001_000000, maxErrors=10
2016-01-14 16:29:59,356 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:29:59,357 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452788999139_0001_000000, appSubmitTime=1452788999257, launchTime=1452788999269
2016-01-14 16:29:59,362 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:29:59,363 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:29:59,365 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:29:59,365 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:29:59,424 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:58623
2016-01-14 16:29:59,442 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 16:29:59,442 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 16:29:59,442 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,464 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:29:59,532 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452788999139_0001_000000, startTime=1452788999532
2016-01-14 16:29:59,533 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:29:59,539 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:29:59,542 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:29:59,560 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:29:59,560 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001, dagName=[BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:29:59,765 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:29:59,874 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452788999139_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/dag_1452788999139_0001_1.dot
2016-01-14 16:29:59,878 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/dag_1452788999139_0001_1-tez-dag.pb.txt
2016-01-14 16:30:00,041 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452788999139_0001_1
2016-01-14 16:30:00,042 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,042 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:30:00 Running Dag: dag_1452788999139_0001_1
2016-01-14 16:30:00,085 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:00,085 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452788999139_0001_1, submitTime=1452788999766
2016-01-14 16:30:00,116 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:00,123 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:00,126 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:00,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452788999139_0001_1, initTime=1452789000087
2016-01-14 16:30:00,128 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:00,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_STARTED]: dagID=dag_1452788999139_0001_1, startTime=1452789000129
2016-01-14 16:30:00,130 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:00,132 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:00,133 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:00,133 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: A7DD97C3B6B2413082CDEDDCDE1B7907 : {9298AB65CE5D47F4A732A1716A77AC67={InputName=9298AB65CE5D47F4A732A1716A77AC67}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:00,135 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 9298AB65CE5D47F4A732A1716A77AC67, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:00,135 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,142 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001, dagName=[BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:30:00,143 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,143 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,145 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]. Starting root input initializers: 1
2016-01-14 16:30:00,150 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 9298AB65CE5D47F4A732A1716A77AC67 on vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,164 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:00,165 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,177 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452788999139_0001, with dag id: dag_1452788999139_0001_1
2016-01-14 16:30:00,182 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:00,184 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 9298AB65CE5D47F4A732A1716A77AC67 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:00,185 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:00,186 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:00,196 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,196 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,233 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,238 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,239 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, vertexId=vertex_1452788999139_0001_1_01, initRequestedTime=1452789000164, initedTime=1452789000236, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,239 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,243 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,258 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:00,262 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 9298AB65CE5D47F4A732A1716A77AC67 on vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,269 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,269 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,270 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,270 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, vertexId=vertex_1452788999139_0001_1_02, initRequestedTime=1452789000243, initedTime=1452789000270, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,271 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,272 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:00,272 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {9298AB65CE5D47F4A732A1716A77AC67=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:00,272 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] parallelism set to 1
2016-01-14 16:30:00,273 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:00,282 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_00, eventCount=1
2016-01-14 16:30:00,287 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,287 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, vertexId=vertex_1452788999139_0001_1_00, initRequestedTime=1452789000133, initedTime=1452789000283, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:00,288 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] with distanceFromRoot: 0
2016-01-14 16:30:00,288 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:00,288 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,317 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,317 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:00,321 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,339 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=085EFCBD350A4927B7CA1ABBE36055AC, vertexName=E37A8363570048DA9BBF994AFA44BDD3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:00,353 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,357 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,357 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, vertexId=vertex_1452788999139_0001_1_03, initRequestedTime=1452789000288, initedTime=1452789000356, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,357 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,359 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:00,360 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,367 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,367 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_00, startRequestedTime=1452789000358, startedTime=1452789000358
2016-01-14 16:30:00,368 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,404 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:00,404 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskId=task_1452788999139_0001_1_00_000000, scheduledTime=1452789000401, launchTime=1452789000401
2016-01-14 16:30:00,404 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:00,405 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_00 for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,405 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] with distanceFromRoot: 1
2016-01-14 16:30:00,407 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,411 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A7DD97C3B6B2413082CDEDDCDE1B7907 in vertex: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,414 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DF8E8CD00C5149F3A7F2E38E12732E8D with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,414 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,414 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_01, startRequestedTime=1452789000407, startedTime=1452789000407
2016-01-14 16:30:00,415 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,417 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A7DD97C3B6B2413082CDEDDCDE1B7907, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,419 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:00,420 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_01 for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,420 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] with distanceFromRoot: 2
2016-01-14 16:30:00,420 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:00,420 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: DF8E8CD00C5149F3A7F2E38E12732E8D in vertex: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:00,421 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: EA7868BACC9E49EA8A22C7E7869174B8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,422 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,422 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_02, startRequestedTime=1452789000420, startedTime=1452789000420
2016-01-14 16:30:00,423 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,423 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_02 for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,423 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] with distanceFromRoot: 3
2016-01-14 16:30:00,423 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: EA7868BACC9E49EA8A22C7E7869174B8 in vertex: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:00,424 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E37A8363570048DA9BBF994AFA44BDD3 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,424 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,425 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_03, startRequestedTime=1452789000423, startedTime=1452789000423
2016-01-14 16:30:00,425 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,468 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:00,472 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:00,473 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:00,480 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000001
2016-01-14 16:30:00,480 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:00,488 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452788999139_0001_00_000001
2016-01-14 16:30:00,488 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:00,496 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:00,496 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:00,496 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000001, launchTime=1452789000496
2016-01-14 16:30:00,500 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,504 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_00_000000_0] to container: [container_1452788999139_0001_00_000001]
2016-01-14 16:30:00,506 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:00,514 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_00_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:00,514 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000001 given task: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,514 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,515 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,515 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,518 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:00,518 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:00,518 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:00,519 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:00,520 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:00,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, startTime=1452789000512, containerId=container_1452788999139_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:00,521 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:00,521 INFO counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters: GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-01-14 16:30:00,529 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:00,543 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:00,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A7DD97C3B6B2413082CDEDDCDE1B7907, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,546 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:00,550 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,552 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:00,552 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,554 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:00,558 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:00,559 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,559 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:00,561 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:00,561 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:00,563 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,567 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,568 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,576 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:00,578 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:00,578 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,587 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,588 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:00,590 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,590 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,608 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, ordinal: 0
2016-01-14 16:30:00,609 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, mem on start (mb), free: 218, total: 281, max: 672
2016-01-14 16:30:00,609 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,609 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:00,610 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:00,610 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:00,612 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:00,616 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:00,617 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:00,617 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:00,618 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:00,620 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:00,621 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 9298AB65CE5D47F4A732A1716A77AC67, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:00,621 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DF8E8CD00C5149F3A7F2E38E12732E8D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:00,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:00,623 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 9298AB65CE5D47F4A732A1716A77AC67 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:00,623 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,624 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:00,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:00,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:00,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:00,628 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,631 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:00,641 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@35097440
2016-01-14 16:30:00,653 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@35097440
2016-01-14 16:30:00,655 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:00,655 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:00,666 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:00,698 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,698 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:00,699 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:00,716 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@645abf3c; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@1c12f9d; comparator=cascading.tuple.hadoop.util.TupleComparator@321ced51; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:00,719 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:00,721 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:00,721 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:00,839 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:00,839 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:00,840 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:00,840 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:00,840 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:00,846 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:00,847 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:00,849 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, all 1 inputs ready in: 00:00:00.002
2016-01-14 16:30:00,860 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,860 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, mem on close (mb), free: 107, total: 281, max: 672
2016-01-14 16:30:00,860 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:00,861 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:00,861 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:00,861 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:00,879 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:00 Completed running task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,901 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:00,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_00, eventCount=1
2016-01-14 16:30:00,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:00,907 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:00,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, startTime=1452789000512, finishTime=1452789000903, timeTaken=391, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:00,912 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:00,913 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:00,916 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,916 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:00,917 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,918 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:00,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskId=task_1452788999139_0001_1_00_000000, startTime=1452789000512, finishTime=1452789000920, timeTaken=408, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:00,929 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:00,930 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] : 1
2016-01-14 16:30:00,930 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,938 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:00,946 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, vertexId=vertex_1452788999139_0001_1_00, initRequestedTime=1452789000133, initedTime=1452789000283, startRequestedTime=1452789000358, startedTime=1452789000358, finishTime=1452789000931, timeTaken=573, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789000512, firstTasksToStart=[ task_1452788999139_0001_1_00_000000 ], lastTaskFinishTime=1452789000903, lastTasksToFinish=[ task_1452788999139_0001_1_00_000000 ], minTaskDuration=391, maxTaskDuration=391, avgTaskDuration=391.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_00_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:00,946 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:00,947 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:00,948 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:00,949 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:00,949 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:00,949 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000001, Exiting
2016-01-14 16:30:00,949 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,949 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000001, stoppedTime=1452789000948, exitStatus=0
2016-01-14 16:30:00,954 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:00,954 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000001 completed successfully
2016-01-14 16:30:00,954 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:00,956 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] attempt: attempt_1452788999139_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:00,957 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,958 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,960 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:00,961 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452788999139_0001_00_000001
2016-01-14 16:30:00,961 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:00,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:00,962 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskId=task_1452788999139_0001_1_01_000000, scheduledTime=1452789000962, launchTime=1452789000962
2016-01-14 16:30:00,963 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:00,964 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,965 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: DF8E8CD00C5149F3A7F2E38E12732E8D, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,965 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:00,966 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:00,967 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000002
2016-01-14 16:30:00,968 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:00,971 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452788999139_0001_00_000002
2016-01-14 16:30:00,971 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:00,975 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:00,976 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000002, launchTime=1452789000975
2016-01-14 16:30:00,976 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:00,977 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000002
2016-01-14 16:30:00,979 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_01_000000_0] to container: [container_1452788999139_0001_00_000002]
2016-01-14 16:30:00,979 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:00,980 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000002 given task: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,980 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,981 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_01_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:00,981 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,982 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,982 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:00,982 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:00,982 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:00,983 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:00,983 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:00,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: DF8E8CD00C5149F3A7F2E38E12732E8D, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,984 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:00,986 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:00,986 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,986 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, startTime=1452789000980, containerId=container_1452788999139_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:00,988 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:00,988 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:00,989 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:00,988 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:00,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,992 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,992 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:00,993 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:00,993 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:00,993 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:00,995 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:00,993 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,002 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:01,002 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,008 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,009 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,009 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,010 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,011 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,025 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, ordinal: 1
2016-01-14 16:30:01,025 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, mem on start (mb), free: 95, total: 281, max: 672
2016-01-14 16:30:01,025 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,025 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,025 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,025 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,026 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,026 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:01,026 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,027 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,027 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:01,027 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, EA7868BACC9E49EA8A22C7E7869174B8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:01,027 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A7DD97C3B6B2413082CDEDDCDE1B7907, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:01,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A7DD97C3B6B2413082CDEDDCDE1B7907 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,038 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,040 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,044 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A7DD97C3B6B2413082CDEDDCDE1B7907 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,046 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:01,046 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,051 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,054 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,054 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,054 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,058 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 16:30:01,064 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_00_000000_0_10002, runDuration: 315011, ]
2016-01-14 16:30:01,064 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:01,066 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,066 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,068 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@55c00779; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@322ac4ea; comparator=cascading.tuple.hadoop.util.TupleComparator@15e5f861; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:01,069 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:01,069 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:01,070 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,072 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_00_000000_0_10002] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789001072, TimeTaken=3, Rate=0.35 MB/s
2016-01-14 16:30:01,139 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,138 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:01,140 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:01,140 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:01,140 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:01,140 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:01,141 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A7DD97C3B6B2413082CDEDDCDE1B7907] #1 in 72ms
2016-01-14 16:30:01,141 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:01,142 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A7DD97C3B6B2413082CDEDDCDE1B7907] #1, status:false, isInterrupted:false
2016-01-14 16:30:01,143 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,144 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 790B4DDE3C8A434C8B2994E08F37E4F6, for 1 inputs
2016-01-14 16:30:01,146 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:01,146 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,150 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,151 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:01,151 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:01,154 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:01,154 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,155 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,158 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:01,161 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@325c4a21; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4906d65b
2016-01-14 16:30:01,212 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,212 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, mem on close (mb), free: 87, total: 345, max: 672
2016-01-14 16:30:01,212 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,213 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,213 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,213 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,214 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,214 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01,215 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:01,215 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:01,215 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:01,216 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:01,252 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,257 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,258 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_01, eventCount=1
2016-01-14 16:30:01,258 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,262 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13]]
2016-01-14 16:30:01,263 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:01,263 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,263 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,265 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, startTime=1452789000980, finishTime=1452789001258, timeTaken=278, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13
2016-01-14 16:30:01,266 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,267 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,267 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,268 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskId=task_1452788999139_0001_1_01_000000, startTime=1452789000980, finishTime=1452789001267, timeTaken=287, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13
2016-01-14 16:30:01,271 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,271 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] : 1
2016-01-14 16:30:01,271 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:01,273 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,279 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, vertexId=vertex_1452788999139_0001_1_01, initRequestedTime=1452789000164, initedTime=1452789000236, startRequestedTime=1452789000407, startedTime=1452789000407, finishTime=1452789001272, timeTaken=865, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13, vertexStats=firstTaskStartTime=1452789000980, firstTasksToStart=[ task_1452788999139_0001_1_01_000000 ], lastTaskFinishTime=1452789001258, lastTasksToFinish=[ task_1452788999139_0001_1_01_000000 ], minTaskDuration=278, maxTaskDuration=278, avgTaskDuration=278.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_01_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,280 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,280 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,281 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000002, stoppedTime=1452789001280, exitStatus=0
2016-01-14 16:30:01,281 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,282 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] attempt: attempt_1452788999139_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:01,281 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,283 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,284 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,284 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,285 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452788999139_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:01,286 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:01,287 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskId=task_1452788999139_0001_1_02_000000, scheduledTime=1452789001286, launchTime=1452789001286
2016-01-14 16:30:01,287 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:01,287 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:01,288 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,288 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EA7868BACC9E49EA8A22C7E7869174B8, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:01,289 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:01,289 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:01,290 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000003
2016-01-14 16:30:01,291 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:01,292 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452788999139_0001_00_000003
2016-01-14 16:30:01,292 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:01,293 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:01,293 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:01,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000003, launchTime=1452789001293
2016-01-14 16:30:01,297 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,298 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,299 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000002: 32 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,299 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000002, Exiting
2016-01-14 16:30:01,299 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,299 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,302 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,302 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_02_000000_0] to container: [container_1452788999139_0001_00_000003]
2016-01-14 16:30:01,302 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000002 completed successfully
2016-01-14 16:30:01,303 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:01,303 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01,303 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000003 given task: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,304 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,304 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_02_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:01,305 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:01,305 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, startTime=1452789001304, containerId=container_1452788999139_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:01,305 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:01,306 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,307 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,308 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,309 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:01,309 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:01,310 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:01,310 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:01,311 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:01,311 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EA7868BACC9E49EA8A22C7E7869174B8, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:01,312 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:01,313 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:01,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,315 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:01,316 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:01,316 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:01,316 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,318 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:01,316 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:01,318 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,318 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,321 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:01,321 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,321 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,322 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,322 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,322 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,326 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,337 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, ordinal: 2
2016-01-14 16:30:01,337 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, mem on start (mb), free: 80, total: 345, max: 672
2016-01-14 16:30:01,337 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,338 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,338 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,339 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,339 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,339 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:01,340 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,340 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,341 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:01,341 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E37A8363570048DA9BBF994AFA44BDD3, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:01,341 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DF8E8CD00C5149F3A7F2E38E12732E8D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:01,342 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,342 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DF8E8CD00C5149F3A7F2E38E12732E8D being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,342 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,343 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,342 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,346 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,347 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,347 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: DF8E8CD00C5149F3A7F2E38E12732E8D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,348 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:01,348 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,349 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,351 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,351 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,353 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,353 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_01_000000_0_10005, runDuration: 256187, ]
2016-01-14 16:30:01,357 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_01_000000_0_10005] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789001356, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:01,358 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,358 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:01,358 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [DF8E8CD00C5149F3A7F2E38E12732E8D] #2 in 3ms
2016-01-14 16:30:01,360 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [DF8E8CD00C5149F3A7F2E38E12732E8D] #2, status:false, isInterrupted:false
2016-01-14 16:30:01,361 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:01,361 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,361 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:01,362 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,362 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,363 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:01,363 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,364 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,365 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,365 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,366 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,367 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@70418842; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@5e58dea1; comparator=cascading.tuple.hadoop.util.TupleComparator@1c804c6b; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:01,367 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:01,368 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:01,368 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,437 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:01,438 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:01,438 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:01,438 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:01,438 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:01,439 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 1C61807F015844DEB80C5A0FA6B960BE, for 1 inputs
2016-01-14 16:30:01,439 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:01,440 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@58416461; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6aff5bd4
2016-01-14 16:30:01,467 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,468 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, mem on close (mb), free: 76, total: 408, max: 672
2016-01-14 16:30:01,468 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,469 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,469 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,470 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,470 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,470 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01,471 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:01,471 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:01,472 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 16:30:01,472 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:01,481 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,487 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,488 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_02, eventCount=1
2016-01-14 16:30:01,489 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,491 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, startTime=1452789001304, finishTime=1452789001488, timeTaken=184, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 16:30:01,492 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,492 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,492 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,493 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskId=task_1452788999139_0001_1_02_000000, startTime=1452789001304, finishTime=1452789001492, timeTaken=188, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 16:30:01,494 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,494 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] : 1
2016-01-14 16:30:01,494 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,497 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 16:30:01,498 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:01,498 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, vertexId=vertex_1452788999139_0001_1_02, initRequestedTime=1452789000243, initedTime=1452789000270, startRequestedTime=1452789000420, startedTime=1452789000420, finishTime=1452789001494, timeTaken=1074, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452789001304, firstTasksToStart=[ task_1452788999139_0001_1_02_000000 ], lastTaskFinishTime=1452789001488, lastTasksToFinish=[ task_1452788999139_0001_1_02_000000 ], minTaskDuration=184, maxTaskDuration=184, avgTaskDuration=184.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_02_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,500 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,497 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,500 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,501 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000003, stoppedTime=1452789001501, exitStatus=0
2016-01-14 16:30:01,501 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,501 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] attempt: attempt_1452788999139_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:01,501 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,502 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,502 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,502 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,502 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452788999139_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:01,502 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,503 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,503 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000003, Exiting
2016-01-14 16:30:01,503 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskId=task_1452788999139_0001_1_03_000000, scheduledTime=1452789001503, launchTime=1452789001503
2016-01-14 16:30:01,503 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,503 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:01,504 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,504 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:01,505 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000003 completed successfully
2016-01-14 16:30:01,507 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:01,507 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,512 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01,512 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E37A8363570048DA9BBF994AFA44BDD3, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:01,512 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:01,512 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:01,513 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000004
2016-01-14 16:30:01,513 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:01,515 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452788999139_0001_00_000004
2016-01-14 16:30:01,516 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:01,518 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:01,518 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000004, launchTime=1452789001518
2016-01-14 16:30:01,518 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:01,519 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,520 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_03_000000_0] to container: [container_1452788999139_0001_00_000004]
2016-01-14 16:30:01,520 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:01,520 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000004 given task: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,520 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,521 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_03_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:01,521 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,521 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,521 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:01,522 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:01,522 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:01,522 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:01,522 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:01,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E37A8363570048DA9BBF994AFA44BDD3, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:01,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, startTime=1452789001520, containerId=container_1452788999139_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:01,523 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:01,524 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:01,521 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:01,522 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:01,527 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:01,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,528 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:01,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:01,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:01,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:01,532 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,534 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,534 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,535 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,536 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,537 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:01,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,538 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:01,538 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,561 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, ordinal: 3
2016-01-14 16:30:01,561 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, mem on start (mb), free: 69, total: 408, max: 672
2016-01-14 16:30:01,562 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,562 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,562 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,570 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:01,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,571 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,571 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,571 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:01,571 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:01,571 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,572 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:01,572 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, EA7868BACC9E49EA8A22C7E7869174B8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:01,572 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 085EFCBD350A4927B7CA1ABBE36055AC, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:01,572 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,572 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: EA7868BACC9E49EA8A22C7E7869174B8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,573 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,573 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,573 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,575 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,575 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,575 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: EA7868BACC9E49EA8A22C7E7869174B8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,576 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:01,576 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,576 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,577 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,579 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,580 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,583 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_02_000000_0_10009, runDuration: 166032, ]
2016-01-14 16:30:01,586 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-01-14 16:30:01,588 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_02_000000_0_10009] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452789001588, TimeTaken=3, Rate=1.38 MB/s
2016-01-14 16:30:01,588 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,589 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,589 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:01,589 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,590 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 16:30:01,589 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [EA7868BACC9E49EA8A22C7E7869174B8] #3 in 4ms
2016-01-14 16:30:01,591 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [EA7868BACC9E49EA8A22C7E7869174B8] #3, status:false, isInterrupted:false
2016-01-14 16:30:01,591 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 47AC7260A856421A87B882BB7B76D34B, for 1 inputs
2016-01-14 16:30:01,592 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,593 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 16:30:01,593 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,593 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,594 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 16:30:01,594 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,594 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,595 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, all 1 inputs ready in: 00:00:00.003
2016-01-14 16:30:01,596 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1a08caf5; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3744a82d
2016-01-14 16:30:01,612 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:01,612 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:01,613 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,613 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452788999139_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:01,614 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,614 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, mem on close (mb), free: 65, total: 408, max: 672
2016-01-14 16:30:01,615 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,615 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,616 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,616 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,616 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,616 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,617 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,619 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:01,619 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,620 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, startTime=1452789001520, finishTime=1452789001618, timeTaken=98, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:01,621 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,621 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,621 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:01,622 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,621 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,622 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,623 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,625 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskId=task_1452788999139_0001_1_03_000000, startTime=1452789001520, finishTime=1452789001622, timeTaken=102, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:01,625 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,626 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] : 1
2016-01-14 16:30:01,626 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,627 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,628 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, vertexId=vertex_1452788999139_0001_1_03, initRequestedTime=1452789000288, initedTime=1452789000356, startRequestedTime=1452789000423, startedTime=1452789000423, finishTime=1452789001626, timeTaken=1203, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789001520, firstTasksToStart=[ task_1452788999139_0001_1_03_000000 ], lastTaskFinishTime=1452789001618, lastTasksToFinish=[ task_1452788999139_0001_1_03_000000 ], minTaskDuration=98, maxTaskDuration=98, avgTaskDuration=98.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_03_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,629 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,629 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000004, stoppedTime=1452789001630, exitStatus=0
2016-01-14 16:30:01,630 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,631 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,631 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452788999139_0001_1
2016-01-14 16:30:01,631 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,633 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,633 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,633 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000004, Exiting
2016-01-14 16:30:01,633 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,633 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,635 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,635 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000004 completed successfully
2016-01-14 16:30:01,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452788999139_0001_1
2016-01-14 16:30:01,638 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:01,639 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:01,639 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,640 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 085EFCBD350A4927B7CA1ABBE36055AC for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,662 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452788999139_0001_1, queueSize=0
2016-01-14 16:30:01,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:01,663 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_FINISHED]: dagId=dag_1452788999139_0001_1, startTime=1452789000129, finishTime=1452789001653, timeTaken=1524, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=16024, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1513619456, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=130, MERGE_PHASE_TIME=149, FIRST_EVENT_RECEIVED=47, LAST_EVENT_RECEIVED=47, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156005042, Process_Duration=111, Process_End_Time=5811156005153, Read_Duration=21, Tuples_Read=303, Tuples_Written=317, Write_Duration=22, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:01,665 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452788999139_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:01,665 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:01,665 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452788999139_0001_00_000004
2016-01-14 16:30:01,666 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01 Completed Dag: dag_1452788999139_0001_1
2016-01-14 16:30:01,667 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452788999139_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:01,667 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452788999139_0001_1_post
2016-01-14 16:30:01,667 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,668 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:01,679 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001
2016-01-14 16:30:01,682 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:01,683 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:01,683 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:01,683 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:01,683 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:01,684 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:01,687 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:01,687 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:01,687 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:01,687 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:01,688 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:01,688 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:01,688 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:01,696 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001
2016-01-14 16:30:01,697 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:02.788
2016-01-14 16:30:01,838 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 16:30:01,876 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:01,876 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:01,982 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.072
2016-01-14 16:30:02,009 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.099
2016-01-14 16:30:02,010 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:02,011 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:02,011 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:02,037 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:02,073 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 16:30:02,258 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting
2016-01-14 16:30:02,259 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,259 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 16:30:02,260 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] parallel execution of steps is enabled: false
2016-01-14 16:30:02,261 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executing total steps: 1
2016-01-14 16:30:02,261 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] allocating management threads: 1
2016-01-14 16:30:02,263 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,263 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:02,282 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tez session mode enabled: true
2016-01-14 16:30:02,288 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:02,288 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:02,288 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:02,300 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001 doesn't exist and is created
2016-01-14 16:30:02,323 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:02,323 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:02,324 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001
2016-01-14 16:30:02,329 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789002288_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:02,343 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:02,344 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:02,346 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:02,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:02,346 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir
2016-01-14 16:30:02,346 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/history.txt.appattempt_1452789002288_0001_000000, maxErrors=10
2016-01-14 16:30:02,347 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:02,347 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789002288_0001_000000, appSubmitTime=1452789002329, launchTime=1452789002329
2016-01-14 16:30:02,351 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:02,354 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:02,355 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:36788
2016-01-14 16:30:02,360 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:02,360 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:02,362 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:02,364 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:02,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789002288_0001_000000, startTime=1452789002364
2016-01-14 16:30:02,364 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:02,424 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:02,424 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:02,424 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001, dagName=[85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,449 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,453 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789002288_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/dag_1452789002288_0001_1.dot
2016-01-14 16:30:02,455 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/dag_1452789002288_0001_1-tez-dag.pb.txt
2016-01-14 16:30:02,501 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789002288_0001_1
2016-01-14 16:30:02,501 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,501 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02 Running Dag: dag_1452789002288_0001_1
2016-01-14 16:30:02,521 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:02,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789002288_0001_1, submitTime=1452789002449
2016-01-14 16:30:02,523 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,523 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:02,523 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:02,524 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789002288_0001_1, initTime=1452789002521
2016-01-14 16:30:02,524 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:02,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_STARTED]: dagID=dag_1452789002288_0001_1, startTime=1452789002524
2016-01-14 16:30:02,525 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:02,525 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:02,525 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:02,525 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: A70FD1D546294A2AB0C612D3B44665F7 : {C900CBB86FF24138ADC3D29199F1C209={InputName=C900CBB86FF24138ADC3D29199F1C209}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:02,525 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: C900CBB86FF24138ADC3D29199F1C209, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:02,526 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001, dagName=[85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,527 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,527 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,528 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,528 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]. Starting root input initializers: 1
2016-01-14 16:30:02,528 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:02,528 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: C900CBB86FF24138ADC3D29199F1C209 on vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,529 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,532 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:02,532 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input C900CBB86FF24138ADC3D29199F1C209 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:02,534 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:02,534 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:02,534 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,535 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,535 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,536 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,536 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, vertexId=vertex_1452789002288_0001_1_01, initRequestedTime=1452789002529, initedTime=1452789002535, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,536 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,536 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,538 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:02,539 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: C900CBB86FF24138ADC3D29199F1C209 on vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,542 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,542 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,542 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,542 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,542 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, vertexId=vertex_1452789002288_0001_1_02, initRequestedTime=1452789002536, initedTime=1452789002542, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,543 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,543 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:02,543 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {C900CBB86FF24138ADC3D29199F1C209=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:02,543 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] parallelism set to 1
2016-01-14 16:30:02,544 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:02,545 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_00, eventCount=1
2016-01-14 16:30:02,547 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, vertexId=vertex_1452789002288_0001_1_00, initRequestedTime=1452789002525, initedTime=1452789002545, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:02,547 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,547 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] with distanceFromRoot: 0
2016-01-14 16:30:02,548 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:02,548 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,551 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] submitted tez dag to app master: application_1452789002288_0001, with dag id: dag_1452789002288_0001_1
2016-01-14 16:30:02,577 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,577 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:02,578 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,583 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=668F7BA666CF49C2884A386FFBEAC8AF, vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:02,583 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, vertexId=vertex_1452789002288_0001_1_03, initRequestedTime=1452789002548, initedTime=1452789002585, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,585 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,585 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,585 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,586 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,586 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_00, startRequestedTime=1452789002585, startedTime=1452789002585
2016-01-14 16:30:02,587 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,589 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,589 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskId=task_1452789002288_0001_1_00_000000, scheduledTime=1452789002588, launchTime=1452789002588
2016-01-14 16:30:02,590 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,590 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_00 for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,591 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] with distanceFromRoot: 1
2016-01-14 16:30:02,592 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,593 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A70FD1D546294A2AB0C612D3B44665F7 in vertex: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,593 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C6F906C21EA04AB8AEC5AF372B2F50C8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,593 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,594 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_01, startRequestedTime=1452789002593, startedTime=1452789002593
2016-01-14 16:30:02,594 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,595 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: A70FD1D546294A2AB0C612D3B44665F7, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,595 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,595 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_01 for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,596 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] with distanceFromRoot: 2
2016-01-14 16:30:02,596 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,596 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C6F906C21EA04AB8AEC5AF372B2F50C8 in vertex: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,597 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: ECA94FF0DA97475897C9BC9E720E2D3C with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,597 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:02,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_02, startRequestedTime=1452789002596, startedTime=1452789002596
2016-01-14 16:30:02,597 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,598 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,599 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:02,599 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:02,600 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000001
2016-01-14 16:30:02,600 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,601 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_02 for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,601 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] with distanceFromRoot: 3
2016-01-14 16:30:02,603 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: ECA94FF0DA97475897C9BC9E720E2D3C in vertex: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,604 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DDAF1DA7D9CA4B3EBC36522E39432376 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,604 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,604 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_03, startRequestedTime=1452789002603, startedTime=1452789002603
2016-01-14 16:30:02,604 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,604 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789002288_0001_00_000001
2016-01-14 16:30:02,604 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:02,607 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,607 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000001, launchTime=1452789002606
2016-01-14 16:30:02,607 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,608 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,609 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_00_000000_0] to container: [container_1452789002288_0001_00_000001]
2016-01-14 16:30:02,609 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,609 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000001 given task: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,609 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,609 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_00_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,610 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, startTime=1452789002609, containerId=container_1452789002288_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:02,610 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,611 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,611 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,615 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,615 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,615 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,615 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,615 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,616 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,616 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: A70FD1D546294A2AB0C612D3B44665F7, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,616 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,619 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:02,621 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:02,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,622 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,622 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,627 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,627 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,627 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:02,628 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,629 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,643 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, ordinal: 0
2016-01-14 16:30:02,643 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, mem on start (mb), free: 254, total: 516, max: 672
2016-01-14 16:30:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,644 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,644 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,644 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:02,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:02,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:02,645 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C900CBB86FF24138ADC3D29199F1C209, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:02,646 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C6F906C21EA04AB8AEC5AF372B2F50C8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C900CBB86FF24138ADC3D29199F1C209 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,646 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:02,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,648 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,649 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,649 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:02,650 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@432659c7
2016-01-14 16:30:02,652 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@432659c7
2016-01-14 16:30:02,652 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:02,652 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:02,655 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,656 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,657 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,657 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,658 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@49e1985a; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@79d9629a; comparator=cascading.tuple.hadoop.util.TupleComparator@2ed8aaf5; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,659 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,659 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,659 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,676 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:02,677 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:02,677 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:02,678 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:02,678 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:02,680 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,680 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:02,681 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:02,686 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,686 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, mem on close (mb), free: 151, total: 516, max: 672
2016-01-14 16:30:02,686 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:02,687 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:02,687 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:02,687 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:02,692 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,697 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:02,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_00, eventCount=1
2016-01-14 16:30:02,698 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:02,699 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:02,699 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:02,700 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,701 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,701 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, startTime=1452789002609, finishTime=1452789002699, timeTaken=90, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:02,702 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:02,702 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:02,702 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,703 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:02,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskId=task_1452789002288_0001_1_00_000000, startTime=1452789002609, finishTime=1452789002703, timeTaken=94, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:02,704 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:02,704 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] : 1
2016-01-14 16:30:02,704 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,706 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:02,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, vertexId=vertex_1452789002288_0001_1_00, initRequestedTime=1452789002525, initedTime=1452789002545, startRequestedTime=1452789002585, startedTime=1452789002585, finishTime=1452789002704, timeTaken=119, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789002609, firstTasksToStart=[ task_1452789002288_0001_1_00_000000 ], lastTaskFinishTime=1452789002699, lastTasksToFinish=[ task_1452789002288_0001_1_00_000000 ], minTaskDuration=90, maxTaskDuration=90, avgTaskDuration=90.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_00_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:02,708 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:02,708 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:02,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000001, stoppedTime=1452789002709, exitStatus=0
2016-01-14 16:30:02,709 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:02,709 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:02,710 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] attempt: attempt_1452789002288_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:02,710 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,710 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,711 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:02,711 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:02,711 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,711 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskId=task_1452789002288_0001_1_01_000000, scheduledTime=1452789002711, launchTime=1452789002711
2016-01-14 16:30:02,712 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,712 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:02,712 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,712 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:02,713 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: C6F906C21EA04AB8AEC5AF372B2F50C8, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,713 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,713 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,713 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:02,715 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000001, Exiting
2016-01-14 16:30:02,715 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,716 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000002
2016-01-14 16:30:02,716 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,718 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,719 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:02,721 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789002288_0001_00_000002
2016-01-14 16:30:02,722 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000001 completed successfully
2016-01-14 16:30:02,723 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:02,723 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:02,726 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,726 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000002, launchTime=1452789002726
2016-01-14 16:30:02,727 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,729 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_01_000000_0] to container: [container_1452789002288_0001_00_000002]
2016-01-14 16:30:02,730 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,730 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_01_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,731 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, startTime=1452789002730, containerId=container_1452789002288_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:02,731 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,731 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02,731 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000002 given task: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,733 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,733 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,733 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,733 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,733 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,733 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: C6F906C21EA04AB8AEC5AF372B2F50C8, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,734 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,734 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,737 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,738 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:02,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,741 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,742 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,744 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:02,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,748 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,749 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:02,749 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:02,749 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,750 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,751 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,751 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,752 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,761 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, ordinal: 1
2016-01-14 16:30:02,761 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, mem on start (mb), free: 144, total: 516, max: 672
2016-01-14 16:30:02,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,762 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,763 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:02,763 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,763 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,764 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:02,764 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A70FD1D546294A2AB0C612D3B44665F7, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:02,764 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, ECA94FF0DA97475897C9BC9E720E2D3C, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:02,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A70FD1D546294A2AB0C612D3B44665F7 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,766 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,767 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,767 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:02,767 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A70FD1D546294A2AB0C612D3B44665F7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:02,768 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:02,768 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:02,768 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:02,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,770 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,771 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_00_000000_0_10015, runDuration: 71633, ]
2016-01-14 16:30:02,772 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_00_000000_0_10015] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789002772, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 16:30:02,772 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,773 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:02,773 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A70FD1D546294A2AB0C612D3B44665F7] #4 in 2ms
2016-01-14 16:30:02,773 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A70FD1D546294A2AB0C612D3B44665F7] #4, status:false, isInterrupted:false
2016-01-14 16:30:02,774 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:02,775 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:02,775 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,775 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:02,776 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:02,776 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,776 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:02,780 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 16:30:02,780 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,780 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,781 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,782 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@40403a1e; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@38a7a886; comparator=cascading.tuple.hadoop.util.TupleComparator@7291279; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,783 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,783 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,783 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,851 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:02,851 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:02,851 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:02,851 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:02,852 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:02,852 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] E45EB07641724802B9BBF837A8D6493F, for 1 inputs
2016-01-14 16:30:02,853 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:02,853 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:02,854 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:02,854 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@56f35464; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1299dd5
2016-01-14 16:30:02,883 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,883 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, mem on close (mb), free: 75, total: 516, max: 672
2016-01-14 16:30:02,884 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,884 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:02,884 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:02,884 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,884 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:02,884 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:02,885 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:02,885 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:02,885 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:02,885 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:02,905 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,909 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:02,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_01, eventCount=1
2016-01-14 16:30:02,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5]]
2016-01-14 16:30:02,911 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:02,911 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:02,912 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, startTime=1452789002730, finishTime=1452789002911, timeTaken=181, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5
2016-01-14 16:30:02,912 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:02,912 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,913 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,912 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:02,914 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,914 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:02,914 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskId=task_1452789002288_0001_1_01_000000, startTime=1452789002730, finishTime=1452789002912, timeTaken=182, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5
2016-01-14 16:30:02,915 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:02,915 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] : 1
2016-01-14 16:30:02,915 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,916 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:02,917 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, vertexId=vertex_1452789002288_0001_1_01, initRequestedTime=1452789002529, initedTime=1452789002535, startRequestedTime=1452789002593, startedTime=1452789002593, finishTime=1452789002916, timeTaken=323, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5, vertexStats=firstTaskStartTime=1452789002730, firstTasksToStart=[ task_1452789002288_0001_1_01_000000 ], lastTaskFinishTime=1452789002911, lastTasksToFinish=[ task_1452789002288_0001_1_01_000000 ], minTaskDuration=181, maxTaskDuration=181, avgTaskDuration=181.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_01_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:02,917 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:02,918 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:02,918 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000002, stoppedTime=1452789002918, exitStatus=0
2016-01-14 16:30:02,918 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:02,918 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] attempt: attempt_1452789002288_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:02,919 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,919 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,919 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:02,918 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:02,919 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:02,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskId=task_1452789002288_0001_1_02_000000, scheduledTime=1452789002920, launchTime=1452789002920
2016-01-14 16:30:02,920 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,921 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,921 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:02,921 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: ECA94FF0DA97475897C9BC9E720E2D3C, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,921 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,922 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,922 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000003
2016-01-14 16:30:02,923 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,924 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789002288_0001_00_000003
2016-01-14 16:30:02,924 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:02,925 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,926 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:02,926 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:02,926 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000002, Exiting
2016-01-14 16:30:02,926 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,926 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,926 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,928 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000003, launchTime=1452789002925
2016-01-14 16:30:02,928 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:02,928 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000002 completed successfully
2016-01-14 16:30:02,929 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000003
2016-01-14 16:30:02,929 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:02,931 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_02_000000_0] to container: [container_1452789002288_0001_00_000003]
2016-01-14 16:30:02,932 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,932 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_02_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,932 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,932 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, startTime=1452789002932, containerId=container_1452789002288_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:02,933 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,933 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02,933 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000003 given task: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,933 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,934 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,934 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,934 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,934 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,934 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,935 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,935 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,935 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: ECA94FF0DA97475897C9BC9E720E2D3C, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,935 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,936 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:02,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,943 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:02,948 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,948 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,948 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:02,948 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,950 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:02,950 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,949 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,951 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,959 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, ordinal: 2
2016-01-14 16:30:02,959 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, mem on start (mb), free: 68, total: 516, max: 672
2016-01-14 16:30:02,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,960 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,960 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,960 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,960 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,960 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,961 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:02,961 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,961 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:02,961 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DDAF1DA7D9CA4B3EBC36522E39432376, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:02,961 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C6F906C21EA04AB8AEC5AF372B2F50C8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:02,961 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C6F906C21EA04AB8AEC5AF372B2F50C8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,965 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,966 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:02,967 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C6F906C21EA04AB8AEC5AF372B2F50C8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:02,967 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:02,967 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:02,967 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:02,968 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,971 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,972 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_01_000000_0_10018, runDuration: 160373, ]
2016-01-14 16:30:02,974 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_01_000000_0_10018] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789002974, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:02,974 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,975 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:02,976 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-01-14 16:30:02,976 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [C6F906C21EA04AB8AEC5AF372B2F50C8] #5 in 3ms
2016-01-14 16:30:02,976 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C6F906C21EA04AB8AEC5AF372B2F50C8] #5, status:false, isInterrupted:false
2016-01-14 16:30:02,977 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:02,977 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:02,976 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,977 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,978 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:02,978 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:02,978 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,978 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:02,979 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:02,979 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:02,983 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@280467e3; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@39407c9b; comparator=cascading.tuple.hadoop.util.TupleComparator@38b4c17a; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,983 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,984 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,984 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,012 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:03,012 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:03,013 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:03,013 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:03,013 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:03,014 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] C08D4F4BD10C479B9A10DF8909280B38, for 1 inputs
2016-01-14 16:30:03,014 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,015 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@52c62ad9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@50273703
2016-01-14 16:30:03,034 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,034 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, mem on close (mb), free: 203, total: 517, max: 672
2016-01-14 16:30:03,034 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:03,034 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:03,035 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:03,035 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,035 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:03,035 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:03,035 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:03,035 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:03,035 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 16:30:03,035 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:03,041 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,044 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:03,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_02, eventCount=1
2016-01-14 16:30:03,046 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:03,047 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, startTime=1452789002932, finishTime=1452789003046, timeTaken=114, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0
2016-01-14 16:30:03,047 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:03,047 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,047 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:03,048 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:03,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskId=task_1452789002288_0001_1_02_000000, startTime=1452789002932, finishTime=1452789003047, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0
2016-01-14 16:30:03,049 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:03,049 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] : 1
2016-01-14 16:30:03,049 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:03,050 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:03,051 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, vertexId=vertex_1452789002288_0001_1_02, initRequestedTime=1452789002536, initedTime=1452789002542, startRequestedTime=1452789002596, startedTime=1452789002596, finishTime=1452789003049, timeTaken=453, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0, vertexStats=firstTaskStartTime=1452789002932, firstTasksToStart=[ task_1452789002288_0001_1_02_000000 ], lastTaskFinishTime=1452789003046, lastTasksToFinish=[ task_1452789002288_0001_1_02_000000 ], minTaskDuration=114, maxTaskDuration=114, avgTaskDuration=114.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_02_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:03,051 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:03,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0]]
2016-01-14 16:30:03,052 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:03,052 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000003, stoppedTime=1452789003052, exitStatus=0
2016-01-14 16:30:03,052 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:03,052 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:03,052 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] attempt: attempt_1452789002288_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:03,052 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:03,053 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,053 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:03,054 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:03,054 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:03,054 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,054 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,055 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:03,055 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskId=task_1452789002288_0001_1_03_000000, scheduledTime=1452789003054, launchTime=1452789003054
2016-01-14 16:30:03,055 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:03,055 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:03,056 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:03,056 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000003, Exiting
2016-01-14 16:30:03,056 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,056 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,056 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:03,057 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,057 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:03,057 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: DDAF1DA7D9CA4B3EBC36522E39432376, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:03,057 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000003 completed successfully
2016-01-14 16:30:03,057 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:03,058 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:03,058 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:03,059 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000004
2016-01-14 16:30:03,059 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:03,061 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789002288_0001_00_000004
2016-01-14 16:30:03,062 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:03,066 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000004, launchTime=1452789003066
2016-01-14 16:30:03,066 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:03,066 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,067 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:03,067 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_03_000000_0] to container: [container_1452789002288_0001_00_000004]
2016-01-14 16:30:03,067 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:03,068 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000004 given task: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,068 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,068 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_03_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:03,069 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,069 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,069 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:03,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, startTime=1452789003068, containerId=container_1452789002288_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:03,069 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:03,070 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:03,070 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:03,070 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:03,070 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:03,070 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: DDAF1DA7D9CA4B3EBC36522E39432376, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:03,070 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:03,071 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:03,071 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:03,072 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:03,072 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:03,073 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,075 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:03,076 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:03,076 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:03,075 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:03,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,077 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:03,078 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,080 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:03,084 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,086 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:03,086 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:03,087 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,100 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:03,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,102 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, ordinal: 3
2016-01-14 16:30:03,102 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, mem on start (mb), free: 194, total: 517, max: 672
2016-01-14 16:30:03,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:03,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:03,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:03,104 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:03,104 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:03,104 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:03,105 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:03,105 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:03,106 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:03,106 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 668F7BA666CF49C2884A386FFBEAC8AF, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:03,106 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, ECA94FF0DA97475897C9BC9E720E2D3C, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:03,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:03,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: ECA94FF0DA97475897C9BC9E720E2D3C being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:03,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:03,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:03,108 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,109 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:03,109 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: ECA94FF0DA97475897C9BC9E720E2D3C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:03,109 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:03,109 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:03,109 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:03,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:03,111 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,112 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_02_000000_0_10021, runDuration: 98756, ]
2016-01-14 16:30:03,114 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-01-14 16:30:03,114 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_02_000000_0_10021] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452789003114, TimeTaken=1, Rate=4.14 MB/s
2016-01-14 16:30:03,114 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,114 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:03,115 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [ECA94FF0DA97475897C9BC9E720E2D3C] #6 in 2ms
2016-01-14 16:30:03,115 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:03,115 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [ECA94FF0DA97475897C9BC9E720E2D3C] #6, status:false, isInterrupted:false
2016-01-14 16:30:03,116 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:03,116 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 16:30:03,116 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,116 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:03,117 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,117 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 16:30:03,117 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,117 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 16:30:03,119 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:03,119 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 309E7276ABA34241BC798F15E92A990D, for 1 inputs
2016-01-14 16:30:03,119 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,120 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@56f10386; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@691e7f01
2016-01-14 16:30:03,138 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:03,138 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:03,139 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,139 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789002288_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:03,140 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,140 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, mem on close (mb), free: 191, total: 517, max: 672
2016-01-14 16:30:03,141 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,141 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:03,142 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:03,142 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,142 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:03,142 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,144 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:03,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:03,146 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:03,146 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, startTime=1452789003068, finishTime=1452789003145, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:03,146 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:03,146 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:03,146 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,146 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,147 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:03,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskId=task_1452789002288_0001_1_03_000000, startTime=1452789003068, finishTime=1452789003146, timeTaken=78, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:03,148 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:03,148 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:03,148 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] : 1
2016-01-14 16:30:03,148 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,149 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:03,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, vertexId=vertex_1452789002288_0001_1_03, initRequestedTime=1452789002548, initedTime=1452789002585, startRequestedTime=1452789002603, startedTime=1452789002603, finishTime=1452789003148, timeTaken=545, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789003068, firstTasksToStart=[ task_1452789002288_0001_1_03_000000 ], lastTaskFinishTime=1452789003145, lastTasksToFinish=[ task_1452789002288_0001_1_03_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_03_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:03,150 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:03,150 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:03,150 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:03,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000004, stoppedTime=1452789003150, exitStatus=0
2016-01-14 16:30:03,151 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:03,151 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:03,151 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789002288_0001_1
2016-01-14 16:30:03,151 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789002288_0001_1
2016-01-14 16:30:03,151 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:03,151 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:03,152 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:03,152 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 668F7BA666CF49C2884A386FFBEAC8AF for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,157 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:03,157 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:03,157 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000004, Exiting
2016-01-14 16:30:03,157 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,157 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,158 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:03,159 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000004 completed successfully
2016-01-14 16:30:03,166 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789002288_0001_1, queueSize=0
2016-01-14 16:30:03,167 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:03,167 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789002288_0001_1, startTime=1452789002524, finishTime=1452789003161, timeTaken=637, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14937, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2167406592, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=20, MERGE_PHASE_TIME=28, FIRST_EVENT_RECEIVED=14, LAST_EVENT_RECEIVED=14, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156011669, Process_Duration=74, Process_End_Time=5811156011743, Read_Duration=21, Tuples_Read=303, Tuples_Written=317, Write_Duration=6, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:03,167 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789002288_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:03,168 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:03,168 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452789002288_0001_00_000004
2016-01-14 16:30:03,169 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:03 Completed Dag: dag_1452789002288_0001_1
2016-01-14 16:30:03,170 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789002288_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:03,170 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789002288_0001_1_post
2016-01-14 16:30:03,170 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,170 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:03,177 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001
2016-01-14 16:30:03,180 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:03,180 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:03,180 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:03,180 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:03,180 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:03,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:03,181 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:03,181 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:03,181 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:03,182 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:03,182 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:03,182 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:03,184 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:03,188 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001
2016-01-14 16:30:03,189 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] completed in: 00:00.930
2016-01-14 16:30:03,270 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 16:30:03,303 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:03,304 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:03,386 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.051
2016-01-14 16:30:03,416 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.080
2016-01-14 16:30:03,417 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:03,417 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:03,417 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:03,440 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:03,475 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 16:30:03,623 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:30:03,623 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,624 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 16:30:03,624 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 16:30:03,624 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 16:30:03,625 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 16:30:03,626 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,626 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:03,642 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:30:03,647 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:03,647 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:03,648 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:03,658 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001 doesn't exist and is created
2016-01-14 16:30:03,677 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:03,677 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:03,678 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001
2016-01-14 16:30:03,683 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789003647_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:03,696 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:03,697 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:03,699 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:03,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:03,700 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir
2016-01-14 16:30:03,700 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/history.txt.appattempt_1452789003647_0001_000000, maxErrors=10
2016-01-14 16:30:03,700 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:03,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789003647_0001_000000, appSubmitTime=1452789003682, launchTime=1452789003682
2016-01-14 16:30:03,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:03,708 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:03,713 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:03,713 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:03,714 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:52802
2016-01-14 16:30:03,717 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:03,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789003647_0001_000000, startTime=1452789003719
2016-01-14 16:30:03,719 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:03,719 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:03,778 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:03,778 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:03,778 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001, dagName=[88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,797 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,799 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789003647_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/dag_1452789003647_0001_1.dot
2016-01-14 16:30:03,801 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/dag_1452789003647_0001_1-tez-dag.pb.txt
2016-01-14 16:30:03,838 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789003647_0001_1
2016-01-14 16:30:03,839 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,839 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03 Running Dag: dag_1452789003647_0001_1
2016-01-14 16:30:03,856 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789003647_0001_1, submitTime=1452789003798
2016-01-14 16:30:03,856 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:03,857 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:03,857 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:03,858 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789003647_0001_1, initTime=1452789003856
2016-01-14 16:30:03,858 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:03,858 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:03,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_STARTED]: dagID=dag_1452789003647_0001_1, startTime=1452789003859
2016-01-14 16:30:03,860 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:03,860 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:03,860 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: ABC4B209A76B4457A915432747B60C7F : {79D4628DF96A475FA05895612AA09065={InputName=79D4628DF96A475FA05895612AA09065}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:03,860 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 79D4628DF96A475FA05895612AA09065, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:03,860 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]. Starting root input initializers: 1
2016-01-14 16:30:03,862 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001, dagName=[88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,862 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:03,862 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 79D4628DF96A475FA05895612AA09065 on vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,862 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,862 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:03,865 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:03,865 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 79D4628DF96A475FA05895612AA09065 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:03,868 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,868 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:03,868 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,869 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,869 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,869 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, vertexId=vertex_1452789003647_0001_1_01, initRequestedTime=1452789003862, initedTime=1452789003869, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,870 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,870 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,868 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:03,875 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:03,875 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,875 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 79D4628DF96A475FA05895612AA09065 on vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,876 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,876 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,876 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,878 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, vertexId=vertex_1452789003647_0001_1_02, initRequestedTime=1452789003870, initedTime=1452789003876, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,878 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,878 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:03,879 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {79D4628DF96A475FA05895612AA09065=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:03,879 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] parallelism set to 1
2016-01-14 16:30:03,879 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:03,880 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_00, eventCount=1
2016-01-14 16:30:03,880 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ABC4B209A76B4457A915432747B60C7F, vertexId=vertex_1452789003647_0001_1_00, initRequestedTime=1452789003860, initedTime=1452789003880, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:03,881 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,881 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] with distanceFromRoot: 0
2016-01-14 16:30:03,881 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:03,881 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,886 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,886 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:03,887 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,888 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452789003647_0001, with dag id: dag_1452789003647_0001_1
2016-01-14 16:30:03,891 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=0BA133967AE44E27A765D92F7FE5973B, vertexName=FDBD1DD030994713ADFA688DAB68291D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:03,891 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, vertexId=vertex_1452789003647_0001_1_03, initRequestedTime=1452789003881, initedTime=1452789003893, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,893 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,893 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,893 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:03,894 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_00, startRequestedTime=1452789003893, startedTime=1452789003893
2016-01-14 16:30:03,894 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,895 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,897 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:03,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskId=task_1452789003647_0001_1_00_000000, scheduledTime=1452789003897, launchTime=1452789003897
2016-01-14 16:30:03,897 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:03,898 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_00 for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,900 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] with distanceFromRoot: 1
2016-01-14 16:30:03,900 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,901 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: ABC4B209A76B4457A915432747B60C7F in vertex: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,901 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B850D941A7724568A46A0CC2FE9491F8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,902 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_01, startRequestedTime=1452789003901, startedTime=1452789003901
2016-01-14 16:30:03,902 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,903 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: ABC4B209A76B4457A915432747B60C7F, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:03,907 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:03,908 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_01 for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,909 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] with distanceFromRoot: 2
2016-01-14 16:30:03,910 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:03,910 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B850D941A7724568A46A0CC2FE9491F8 in vertex: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:03,911 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D0E9805FF68B45CAB52AC3539931AA0A with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,911 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,911 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_02, startRequestedTime=1452789003910, startedTime=1452789003910
2016-01-14 16:30:03,911 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,911 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:03,912 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_02 for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,912 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] with distanceFromRoot: 3
2016-01-14 16:30:03,913 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:03,913 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:03,914 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000001
2016-01-14 16:30:03,915 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:03,915 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D0E9805FF68B45CAB52AC3539931AA0A in vertex: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:03,915 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: FDBD1DD030994713ADFA688DAB68291D with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,915 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,916 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_03, startRequestedTime=1452789003915, startedTime=1452789003915
2016-01-14 16:30:03,916 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,918 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789003647_0001_00_000001
2016-01-14 16:30:03,918 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:03,919 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:03,919 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:03,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000001, launchTime=1452789003919
2016-01-14 16:30:03,921 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000001
2016-01-14 16:30:03,922 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_00_000000_0] to container: [container_1452789003647_0001_00_000001]
2016-01-14 16:30:03,922 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:03,923 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000001 given task: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_00_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,924 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:03,924 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:03,924 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:03,924 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, startTime=1452789003923, containerId=container_1452789003647_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:03,925 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:03,924 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:03,925 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:03,926 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:03,926 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: ABC4B209A76B4457A915432747B60C7F, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:03,927 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:03,925 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:03,928 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,928 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:03,928 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:03,933 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:03,933 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,934 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,934 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:03,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:03,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:03,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,935 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:03,938 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:03,938 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,939 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:03,939 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,949 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ABC4B209A76B4457A915432747B60C7F, ordinal: 0
2016-01-14 16:30:03,949 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABC4B209A76B4457A915432747B60C7F, mem on start (mb), free: 54, total: 517, max: 672
2016-01-14 16:30:03,950 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,950 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:03,950 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:03,950 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:03,950 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:03,951 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:03,951 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:03,951 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:03,952 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:03,952 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:03,952 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 79D4628DF96A475FA05895612AA09065, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:03,953 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B850D941A7724568A46A0CC2FE9491F8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:03,953 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:03,953 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 79D4628DF96A475FA05895612AA09065 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:03,953 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,954 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:03,953 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:03,954 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:03,954 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,955 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:03,955 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,956 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:03,957 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@378095fb
2016-01-14 16:30:03,959 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@378095fb
2016-01-14 16:30:03,959 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:03,960 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:03,961 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,962 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,962 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:03,962 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:03,963 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6d47d006; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@5aa71f7; comparator=cascading.tuple.hadoop.util.TupleComparator@79201208; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:03,964 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:03,964 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:03,964 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,990 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:03,990 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:03,991 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:03,991 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:03,991 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:03,993 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,994 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:03,994 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ABC4B209A76B4457A915432747B60C7F, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,997 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,998 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABC4B209A76B4457A915432747B60C7F, mem on close (mb), free: 161, total: 516, max: 672
2016-01-14 16:30:03,998 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:03,998 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:03,998 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:03,998 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:04,001 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04,007 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_00, eventCount=1
2016-01-14 16:30:04,008 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, startTime=1452789003923, finishTime=1452789004008, timeTaken=85, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:04,008 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,009 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,009 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04,010 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskId=task_1452789003647_0001_1_00_000000, startTime=1452789003923, finishTime=1452789004009, timeTaken=86, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:04,010 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,010 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,010 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] : 1
2016-01-14 16:30:04,010 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:04,011 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,012 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, vertexId=vertex_1452789003647_0001_1_00, initRequestedTime=1452789003860, initedTime=1452789003880, startRequestedTime=1452789003893, startedTime=1452789003893, finishTime=1452789004010, timeTaken=117, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789003923, firstTasksToStart=[ task_1452789003647_0001_1_00_000000 ], lastTaskFinishTime=1452789004008, lastTasksToFinish=[ task_1452789003647_0001_1_00_000000 ], minTaskDuration=85, maxTaskDuration=85, avgTaskDuration=85.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_00_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,012 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,012 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,012 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000001, stoppedTime=1452789004012, exitStatus=0
2016-01-14 16:30:04,013 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,013 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:04,013 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] attempt: attempt_1452789003647_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,012 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,013 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,013 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:04,014 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,014 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,014 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,015 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,015 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,015 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000001, Exiting
2016-01-14 16:30:04,015 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,015 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,015 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,016 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,017 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskId=task_1452789003647_0001_1_01_000000, scheduledTime=1452789004016, launchTime=1452789004016
2016-01-14 16:30:04,017 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,018 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000001 completed successfully
2016-01-14 16:30:04,018 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,018 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,018 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,018 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,019 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B850D941A7724568A46A0CC2FE9491F8, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,019 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,019 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,020 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000002
2016-01-14 16:30:04,020 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,022 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789003647_0001_00_000002
2016-01-14 16:30:04,023 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,025 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,025 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000002, launchTime=1452789004025
2016-01-14 16:30:04,026 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,027 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_01_000000_0] to container: [container_1452789003647_0001_00_000002]
2016-01-14 16:30:04,027 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,027 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000002 given task: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,028 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_01_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,028 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,028 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,028 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, startTime=1452789004028, containerId=container_1452789003647_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:04,029 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,029 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,029 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,030 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,030 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,030 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,030 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,030 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,031 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,031 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B850D941A7724568A46A0CC2FE9491F8, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,031 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,032 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,034 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,034 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:04,036 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:04,037 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,037 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,038 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,038 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,038 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,039 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,040 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:04,040 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,041 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,048 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, ordinal: 1
2016-01-14 16:30:04,048 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, mem on start (mb), free: 153, total: 516, max: 672
2016-01-14 16:30:04,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,048 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,049 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,049 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:04,049 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,049 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:04,050 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, ABC4B209A76B4457A915432747B60C7F, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:04,050 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D0E9805FF68B45CAB52AC3539931AA0A, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:04,050 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: ABC4B209A76B4457A915432747B60C7F being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,053 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,054 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,054 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,055 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: ABC4B209A76B4457A915432747B60C7F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,055 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:04,056 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,056 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,060 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,060 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,061 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,063 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 16:30:04,064 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:04,064 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,064 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,064 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_00_000000_0_10027, runDuration: 68811, ]
2016-01-14 16:30:04,065 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@70e793d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@47399769; comparator=cascading.tuple.hadoop.util.TupleComparator@16039307; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:04,065 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:04,066 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:04,067 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,067 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_00_000000_0_10027] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789004067, TimeTaken=2, Rate=0.52 MB/s
2016-01-14 16:30:04,068 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,068 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:04,068 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [ABC4B209A76B4457A915432747B60C7F] #7 in 3ms
2016-01-14 16:30:04,069 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [ABC4B209A76B4457A915432747B60C7F] #7, status:false, isInterrupted:false
2016-01-14 16:30:04,076 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:04,078 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:04,078 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:04,078 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:04,078 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:04,076 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,079 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:04,079 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0016BEC6C7F04E1DA64780FB4A4C80A3, for 1 inputs
2016-01-14 16:30:04,079 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,079 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:04,079 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:04,079 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,080 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:04,080 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,081 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:04,081 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,082 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3376b54c; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3678aaf9
2016-01-14 16:30:04,111 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,111 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, mem on close (mb), free: 87, total: 516, max: 672
2016-01-14 16:30:04,111 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,111 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,112 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,112 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,112 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,112 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04,112 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:04,113 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:04,113 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:04,113 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:04,127 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,133 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,134 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_01, eventCount=1
2016-01-14 16:30:04,135 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,135 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-01-14 16:30:04,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, startTime=1452789004028, finishTime=1452789004134, timeTaken=106, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 16:30:04,136 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:04,137 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,138 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,138 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,138 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,138 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,138 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskId=task_1452789003647_0001_1_01_000000, startTime=1452789004028, finishTime=1452789004138, timeTaken=110, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 16:30:04,139 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,139 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] : 1
2016-01-14 16:30:04,139 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,141 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, vertexId=vertex_1452789003647_0001_1_01, initRequestedTime=1452789003862, initedTime=1452789003869, startRequestedTime=1452789003901, startedTime=1452789003901, finishTime=1452789004140, timeTaken=239, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1452789004028, firstTasksToStart=[ task_1452789003647_0001_1_01_000000 ], lastTaskFinishTime=1452789004134, lastTasksToFinish=[ task_1452789003647_0001_1_01_000000 ], minTaskDuration=106, maxTaskDuration=106, avgTaskDuration=106.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_01_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,142 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,142 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,142 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] attempt: attempt_1452789003647_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,142 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,142 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,143 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000002, stoppedTime=1452789004143, exitStatus=0
2016-01-14 16:30:04,143 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,143 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,143 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskId=task_1452789003647_0001_1_02_000000, scheduledTime=1452789004144, launchTime=1452789004144
2016-01-14 16:30:04,144 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,144 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,145 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,145 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,146 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D0E9805FF68B45CAB52AC3539931AA0A, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,146 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,147 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,147 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000003
2016-01-14 16:30:04,148 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,150 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789003647_0001_00_000003
2016-01-14 16:30:04,150 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:04,151 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,151 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000002: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,151 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,151 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,153 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000002, Exiting
2016-01-14 16:30:04,153 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,153 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,154 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000003, launchTime=1452789004151
2016-01-14 16:30:04,154 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,154 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000002 completed successfully
2016-01-14 16:30:04,154 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,156 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,157 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_02_000000_0] to container: [container_1452789003647_0001_00_000003]
2016-01-14 16:30:04,157 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,158 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_02_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,159 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, startTime=1452789004158, containerId=container_1452789003647_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:04,159 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,159 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,159 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04,160 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000003 given task: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,161 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,162 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,162 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,162 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,162 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,162 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,163 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,163 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D0E9805FF68B45CAB52AC3539931AA0A, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,163 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,166 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,166 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,166 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,169 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,169 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:04,172 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,172 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,176 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:04,176 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,178 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:04,178 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,179 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,181 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,182 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,182 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,184 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,188 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, ordinal: 2
2016-01-14 16:30:04,188 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, mem on start (mb), free: 81, total: 516, max: 672
2016-01-14 16:30:04,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,188 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,189 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:04,190 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,191 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,191 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:04,192 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, FDBD1DD030994713ADFA688DAB68291D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:04,193 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B850D941A7724568A46A0CC2FE9491F8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:04,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B850D941A7724568A46A0CC2FE9491F8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,195 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,195 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,195 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,196 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,198 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B850D941A7724568A46A0CC2FE9491F8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,198 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:04,198 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,198 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,201 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,201 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,203 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_01_000000_0_10030, runDuration: 91682, ]
2016-01-14 16:30:04,204 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:04,205 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,207 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,207 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,207 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_01_000000_0_10030] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789004207, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:04,209 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,208 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@2eec5c0d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@1f8f718c; comparator=cascading.tuple.hadoop.util.TupleComparator@267ae84; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:04,210 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:04,210 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:04,210 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,209 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:04,221 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [B850D941A7724568A46A0CC2FE9491F8] #8 in 16ms
2016-01-14 16:30:04,221 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B850D941A7724568A46A0CC2FE9491F8] #8, status:false, isInterrupted:false
2016-01-14 16:30:04,229 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:04,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:04,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,231 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,231 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:04,231 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:04,232 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:04,232 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:04,231 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:04,232 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,233 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,234 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 3F18F079E0944129A47CF89B52319255, for 1 inputs
2016-01-14 16:30:04,235 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:04,235 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1984599; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@340140a0
2016-01-14 16:30:04,255 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,256 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, mem on close (mb), free: 200, total: 516, max: 672
2016-01-14 16:30:04,256 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,257 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,257 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,258 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,258 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,258 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04,259 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:04,259 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:04,259 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 16:30:04,260 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:04,265 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,269 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_02, eventCount=1
2016-01-14 16:30:04,271 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, startTime=1452789004158, finishTime=1452789004270, timeTaken=112, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-01-14 16:30:04,272 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,272 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,272 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,273 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskId=task_1452789003647_0001_1_02_000000, startTime=1452789004158, finishTime=1452789004273, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-01-14 16:30:04,274 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,274 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] : 1
2016-01-14 16:30:04,274 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,275 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, vertexId=vertex_1452789003647_0001_1_02, initRequestedTime=1452789003870, initedTime=1452789003876, startRequestedTime=1452789003910, startedTime=1452789003910, finishTime=1452789004274, timeTaken=364, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4, vertexStats=firstTaskStartTime=1452789004158, firstTasksToStart=[ task_1452789003647_0001_1_02_000000 ], lastTaskFinishTime=1452789004270, lastTasksToFinish=[ task_1452789003647_0001_1_02_000000 ], minTaskDuration=112, maxTaskDuration=112, avgTaskDuration=112.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_02_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,276 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,276 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000003, stoppedTime=1452789004276, exitStatus=0
2016-01-14 16:30:04,276 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,276 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,277 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] attempt: attempt_1452789003647_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,277 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,277 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,277 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,277 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,278 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskId=task_1452789003647_0001_1_03_000000, scheduledTime=1452789004278, launchTime=1452789004278
2016-01-14 16:30:04,278 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,278 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4]]
2016-01-14 16:30:04,278 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,279 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,279 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,279 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FDBD1DD030994713ADFA688DAB68291D, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:04,279 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:04,280 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,280 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,279 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,280 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,280 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,280 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,281 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000003, Exiting
2016-01-14 16:30:04,281 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,281 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,281 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,282 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000003 completed successfully
2016-01-14 16:30:04,282 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000004
2016-01-14 16:30:04,282 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,283 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,284 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789003647_0001_00_000004
2016-01-14 16:30:04,285 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,290 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000004, launchTime=1452789004290
2016-01-14 16:30:04,290 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,291 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,292 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,293 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_03_000000_0] to container: [container_1452789003647_0001_00_000004]
2016-01-14 16:30:04,293 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,293 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000004 given task: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,293 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,294 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_03_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,294 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,294 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, startTime=1452789004294, containerId=container_1452789003647_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:04,294 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,294 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,295 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,295 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,295 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,295 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,295 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,296 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FDBD1DD030994713ADFA688DAB68291D, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:04,296 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,297 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04,298 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,298 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,299 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,299 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:04,299 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,301 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,301 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:04,302 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,302 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,303 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:04,303 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,304 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,309 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,309 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,309 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,317 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,319 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:04,319 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,323 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, ordinal: 3
2016-01-14 16:30:04,323 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, mem on start (mb), free: 193, total: 516, max: 672
2016-01-14 16:30:04,323 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,323 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,323 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,324 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,324 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:04,324 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:04,324 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,324 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,325 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:04,325 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 0BA133967AE44E27A765D92F7FE5973B, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:04,325 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D0E9805FF68B45CAB52AC3539931AA0A, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:04,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D0E9805FF68B45CAB52AC3539931AA0A being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,326 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,327 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,327 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D0E9805FF68B45CAB52AC3539931AA0A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,327 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:04,328 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,328 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,328 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,329 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,330 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,332 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_02_000000_0_10033, runDuration: 95380, ]
2016-01-14 16:30:04,334 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 16:30:04,335 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,335 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,335 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_02_000000_0_10033] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452789004335, TimeTaken=2, Rate=2.44 MB/s
2016-01-14 16:30:04,336 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,336 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:04,336 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 16:30:04,336 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D0E9805FF68B45CAB52AC3539931AA0A] #9 in 3ms
2016-01-14 16:30:04,336 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D0E9805FF68B45CAB52AC3539931AA0A] #9, status:false, isInterrupted:false
2016-01-14 16:30:04,337 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 264191978B304D7DA6FEC6B650FF46F5, for 1 inputs
2016-01-14 16:30:04,337 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,338 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 16:30:04,338 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,338 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,338 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 16:30:04,338 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,339 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,339 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, all 1 inputs ready in: 00:00:00.002
2016-01-14 16:30:04,340 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4d7c5890; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1e9a7dae
2016-01-14 16:30:04,348 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:04,348 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:04,349 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,349 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789003647_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:04,350 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,350 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, mem on close (mb), free: 190, total: 516, max: 672
2016-01-14 16:30:04,350 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,351 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,351 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,351 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,351 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,352 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,352 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,353 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:04,354 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, startTime=1452789004294, finishTime=1452789004353, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:04,353 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,355 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:04,355 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,355 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,354 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,356 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,356 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,357 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,357 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskId=task_1452789003647_0001_1_03_000000, startTime=1452789004294, finishTime=1452789004356, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:04,357 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,357 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] : 1
2016-01-14 16:30:04,357 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,358 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,359 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, vertexId=vertex_1452789003647_0001_1_03, initRequestedTime=1452789003881, initedTime=1452789003893, startRequestedTime=1452789003915, startedTime=1452789003915, finishTime=1452789004357, timeTaken=442, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789004294, firstTasksToStart=[ task_1452789003647_0001_1_03_000000 ], lastTaskFinishTime=1452789004353, lastTasksToFinish=[ task_1452789003647_0001_1_03_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_03_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,359 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,359 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,359 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000004, stoppedTime=1452789004359, exitStatus=0
2016-01-14 16:30:04,359 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,359 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,359 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,360 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789003647_0001_1
2016-01-14 16:30:04,360 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789003647_0001_1
2016-01-14 16:30:04,360 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,360 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 0BA133967AE44E27A765D92F7FE5973B for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,366 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,366 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,366 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000004, Exiting
2016-01-14 16:30:04,367 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,367 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,368 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,368 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000004 completed successfully
2016-01-14 16:30:04,370 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,370 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:04,374 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789003647_0001_1, queueSize=0
2016-01-14 16:30:04,374 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:04,374 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789003647_0001_1, startTime=1452789003859, finishTime=1452789004370, timeTaken=511, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15771, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=22, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2165309440, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=47, MERGE_PHASE_TIME=72, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156016649, Process_Duration=64, Process_End_Time=5811156016713, Read_Duration=10, Tuples_Read=303, Tuples_Written=317, Write_Duration=12, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:04,374 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789003647_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:04,375 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:04,375 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452789003647_0001_00_000004
2016-01-14 16:30:04,375 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04 Completed Dag: dag_1452789003647_0001_1
2016-01-14 16:30:04,375 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789003647_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:04,375 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789003647_0001_1_post
2016-01-14 16:30:04,375 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,376 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:04,383 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001
2016-01-14 16:30:04,386 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:04,386 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:04,386 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:04,386 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:04,386 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:04,387 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:04,388 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:04,388 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:04,388 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:04,389 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:04,389 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:04,390 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:04,390 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:04,397 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001
2016-01-14 16:30:04,398 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:00.774
2016-01-14 16:30:04,479 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 16:30:04,510 INFO util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:04,511 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:04,581 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.042
2016-01-14 16:30:04,603 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.064
2016-01-14 16:30:04,604 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:04,604 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:04,604 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:04,624 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:04,647 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-01-14 16:30:04,784 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:30:04,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:04,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 16:30:04,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] parallel execution of steps is enabled: false
2016-01-14 16:30:04,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executing total steps: 1
2016-01-14 16:30:04,785 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] allocating management threads: 1
2016-01-14 16:30:04,786 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,786 WARN flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:04,799 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:30:04,804 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:04,804 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:04,804 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:04,814 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001 doesn't exist and is created
2016-01-14 16:30:04,831 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:04,831 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:04,831 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001
2016-01-14 16:30:04,835 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789004804_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:04,846 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:04,847 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:04,849 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:04,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:04,849 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir
2016-01-14 16:30:04,849 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/history.txt.appattempt_1452789004804_0001_000000, maxErrors=10
2016-01-14 16:30:04,850 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:04,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789004804_0001_000000, appSubmitTime=1452789004835, launchTime=1452789004835
2016-01-14 16:30:04,857 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:04,860 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:04,861 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:51738
2016-01-14 16:30:04,862 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:04,865 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,865 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:04,867 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789004804_0001_000000, startTime=1452789004867
2016-01-14 16:30:04,867 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:04,867 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:04,931 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:04,931 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:04,931 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001, dagName=[033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,950 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,953 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789004804_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/dag_1452789004804_0001_1.dot
2016-01-14 16:30:04,955 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/dag_1452789004804_0001_1-tez-dag.pb.txt
2016-01-14 16:30:04,998 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789004804_0001_1
2016-01-14 16:30:04,998 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,998 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04 Running Dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,013 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789004804_0001_1, submitTime=1452789004950
2016-01-14 16:30:05,013 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:05,014 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,015 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:05,015 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:05,015 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789004804_0001_1, initTime=1452789005013
2016-01-14 16:30:05,016 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:05,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_STARTED]: dagID=dag_1452789004804_0001_1, startTime=1452789005016
2016-01-14 16:30:05,017 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:05,017 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:05,017 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001, dagName=[033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:05,017 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E3B0FB85E18E4F9BA18C85D043EA8509 : {DA81235C682A4596919C8AEC404F7E66={InputName=DA81235C682A4596919C8AEC404F7E66}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: DA81235C682A4596919C8AEC404F7E66, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]. Starting root input initializers: 1
2016-01-14 16:30:05,020 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:05,021 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: DA81235C682A4596919C8AEC404F7E66 on vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,021 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,022 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:05,023 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input DA81235C682A4596919C8AEC404F7E66 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:05,023 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:05,023 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:05,026 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:05,026 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: DA81235C682A4596919C8AEC404F7E66 on vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,028 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,028 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,030 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,030 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,031 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, vertexId=vertex_1452789004804_0001_1_01, initRequestedTime=1452789005021, initedTime=1452789005030, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,031 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,032 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:05,032 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {DA81235C682A4596919C8AEC404F7E66=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:05,033 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] parallelism set to 1
2016-01-14 16:30:05,033 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:05,034 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_00, eventCount=1
2016-01-14 16:30:05,035 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,035 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, vertexId=vertex_1452789004804_0001_1_00, initRequestedTime=1452789005019, initedTime=1452789005034, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:05,036 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] with distanceFromRoot: 0
2016-01-14 16:30:05,036 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:05,036 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452789004804_0001, with dag id: dag_1452789004804_0001_1
2016-01-14 16:30:05,036 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,040 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,041 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,041 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,041 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A016A996D30D455BA1D753AC00B9C434, vertexId=vertex_1452789004804_0001_1_02, initRequestedTime=1452789005036, initedTime=1452789005041, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,043 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,043 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,043 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,044 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_00, startRequestedTime=1452789005043, startedTime=1452789005043
2016-01-14 16:30:05,045 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,045 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,053 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,054 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:05,055 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,057 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:05,057 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,059 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, vertexId=vertex_1452789004804_0001_1_03, initRequestedTime=1452789005045, initedTime=1452789005059, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,059 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,059 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,060 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,061 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskId=task_1452789004804_0001_1_00_000000, scheduledTime=1452789005060, launchTime=1452789005060
2016-01-14 16:30:05,061 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,062 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_00 for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,062 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] with distanceFromRoot: 1
2016-01-14 16:30:05,063 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,063 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E3B0FB85E18E4F9BA18C85D043EA8509 in vertex: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,064 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D2A0861392DE4D708D767DEDBAB90939 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,064 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,064 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_01, startRequestedTime=1452789005063, startedTime=1452789005063
2016-01-14 16:30:05,065 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,065 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: E3B0FB85E18E4F9BA18C85D043EA8509, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,066 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,066 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_01 for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,066 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] with distanceFromRoot: 2
2016-01-14 16:30:05,067 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,067 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D2A0861392DE4D708D767DEDBAB90939 in vertex: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,068 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A016A996D30D455BA1D753AC00B9C434 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,068 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_02, startRequestedTime=1452789005067, startedTime=1452789005067
2016-01-14 16:30:05,069 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,069 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_02 for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,069 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] with distanceFromRoot: 3
2016-01-14 16:30:05,070 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A016A996D30D455BA1D753AC00B9C434 in vertex: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,070 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: AE7FFF9183E44A5CA31F76C68F5532F6 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,071 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_03, startRequestedTime=1452789005070, startedTime=1452789005070
2016-01-14 16:30:05,071 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,072 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:05,072 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:05,073 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:05,073 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000001
2016-01-14 16:30:05,074 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,076 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789004804_0001_00_000001
2016-01-14 16:30:05,076 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:05,079 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,079 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000001, launchTime=1452789005078
2016-01-14 16:30:05,080 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,081 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_00_000000_0] to container: [container_1452789004804_0001_00_000001]
2016-01-14 16:30:05,082 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,082 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_00_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,083 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, startTime=1452789005082, containerId=container_1452789004804_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:05,083 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,083 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,083 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,084 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000001 given task: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,084 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,085 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,085 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,085 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,085 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,086 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,086 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,086 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,086 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: E3B0FB85E18E4F9BA18C85D043EA8509, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,086 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,088 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,090 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:05,090 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:05,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,094 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,099 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,100 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,103 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:05,103 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,104 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,105 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,105 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,109 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, ordinal: 0
2016-01-14 16:30:05,109 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, mem on start (mb), free: 60, total: 516, max: 672
2016-01-14 16:30:05,109 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,110 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,111 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:05,111 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:05,111 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,112 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,112 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:05,112 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DA81235C682A4596919C8AEC404F7E66, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:05,112 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D2A0861392DE4D708D767DEDBAB90939, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:05,113 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,113 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,113 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:05,113 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,114 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DA81235C682A4596919C8AEC404F7E66 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,114 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,115 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,115 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,115 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,116 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:05,116 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@6c03cac5
2016-01-14 16:30:05,118 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@6c03cac5
2016-01-14 16:30:05,118 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:05,118 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:05,120 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:05,121 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,121 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,122 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,122 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6796ffe0; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6a790041; comparator=cascading.tuple.hadoop.util.TupleComparator@2dffe203; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,123 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,123 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,123 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,147 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:05,147 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:05,147 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:05,147 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:05,148 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:05,149 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:05,149 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:05,149 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,153 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,153 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, mem on close (mb), free: 162, total: 517, max: 672
2016-01-14 16:30:05,153 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,154 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,154 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:05,154 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:05,158 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,162 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,163 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_00, eventCount=1
2016-01-14 16:30:05,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:05,163 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, startTime=1452789005082, finishTime=1452789005163, timeTaken=81, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:05,164 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:05,163 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,164 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,164 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,164 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,165 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,165 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,165 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,165 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskId=task_1452789004804_0001_1_00_000000, startTime=1452789005082, finishTime=1452789005165, timeTaken=83, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:05,166 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,166 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] : 1
2016-01-14 16:30:05,166 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,168 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,168 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, vertexId=vertex_1452789004804_0001_1_00, initRequestedTime=1452789005019, initedTime=1452789005034, startRequestedTime=1452789005043, startedTime=1452789005043, finishTime=1452789005166, timeTaken=123, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789005082, firstTasksToStart=[ task_1452789004804_0001_1_00_000000 ], lastTaskFinishTime=1452789005163, lastTasksToFinish=[ task_1452789004804_0001_1_00_000000 ], minTaskDuration=81, maxTaskDuration=81, avgTaskDuration=81.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_00_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,168 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,168 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,168 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] attempt: attempt_1452789004804_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,169 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,169 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,169 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000001, stoppedTime=1452789005169, exitStatus=0
2016-01-14 16:30:05,169 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,170 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,170 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskId=task_1452789004804_0001_1_01_000000, scheduledTime=1452789005170, launchTime=1452789005170
2016-01-14 16:30:05,171 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,171 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,172 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,172 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: D2A0861392DE4D708D767DEDBAB90939, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,172 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,172 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,173 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,173 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,174 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000002
2016-01-14 16:30:05,174 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,176 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789004804_0001_00_000002
2016-01-14 16:30:05,176 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:05,177 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,177 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,178 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,178 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000001: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,178 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000001, Exiting
2016-01-14 16:30:05,178 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,178 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000002, launchTime=1452789005177
2016-01-14 16:30:05,180 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,181 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,181 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000001 completed successfully
2016-01-14 16:30:05,181 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_01_000000_0] to container: [container_1452789004804_0001_00_000002]
2016-01-14 16:30:05,182 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,182 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000002 given task: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,182 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,182 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_01_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,183 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,183 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, startTime=1452789005182, containerId=container_1452789004804_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:05,184 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,184 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,183 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,185 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,185 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,185 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,185 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,185 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: D2A0861392DE4D708D767DEDBAB90939, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,185 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,186 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,186 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,186 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,189 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,191 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:05,192 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,192 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,192 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,191 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,192 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,195 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,195 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,196 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,200 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, ordinal: 1
2016-01-14 16:30:05,200 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, mem on start (mb), free: 155, total: 517, max: 672
2016-01-14 16:30:05,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,201 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,201 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,201 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,201 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,202 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,202 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,202 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:05,202 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A016A996D30D455BA1D753AC00B9C434, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:05,203 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E3B0FB85E18E4F9BA18C85D043EA8509, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:05,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E3B0FB85E18E4F9BA18C85D043EA8509 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,204 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,205 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,205 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,206 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E3B0FB85E18E4F9BA18C85D043EA8509 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,206 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:05,206 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,206 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,207 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,207 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,208 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,209 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_00_000000_0_10039, runDuration: 60175, ]
2016-01-14 16:30:05,210 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 16:30:05,211 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_00_000000_0_10039] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789005211, TimeTaken=0, Rate=0.00 MB/s
2016-01-14 16:30:05,211 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,211 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,211 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:05,211 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,211 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,212 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@25d6868; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@4271850e; comparator=cascading.tuple.hadoop.util.TupleComparator@2ad67c36; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,212 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,212 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,212 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,211 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E3B0FB85E18E4F9BA18C85D043EA8509] #10 in 1ms
2016-01-14 16:30:05,214 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E3B0FB85E18E4F9BA18C85D043EA8509] #10, status:false, isInterrupted:false
2016-01-14 16:30:05,221 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:05,221 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:05,221 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:05,222 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:05,222 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:05,221 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,223 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:05,223 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,223 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,224 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:05,224 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,226 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,226 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 276E68A215354EAC919589E262B6FC62, for 1 inputs
2016-01-14 16:30:05,226 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:05,227 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:05,227 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,228 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@5c2b7b8b; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@2abc11f3
2016-01-14 16:30:05,254 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,254 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, mem on close (mb), free: 89, total: 517, max: 672
2016-01-14 16:30:05,254 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,254 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,254 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,254 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,254 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,256 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05,256 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,256 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,256 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:05,256 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:05,264 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,267 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3]]
2016-01-14 16:30:05,268 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:05,269 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,269 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_01, eventCount=1
2016-01-14 16:30:05,269 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,270 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, startTime=1452789005182, finishTime=1452789005269, timeTaken=87, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3
2016-01-14 16:30:05,270 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,270 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskId=task_1452789004804_0001_1_01_000000, startTime=1452789005182, finishTime=1452789005270, timeTaken=88, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3
2016-01-14 16:30:05,271 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,271 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] : 1
2016-01-14 16:30:05,272 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,271 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,271 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, vertexId=vertex_1452789004804_0001_1_01, initRequestedTime=1452789005021, initedTime=1452789005030, startRequestedTime=1452789005063, startedTime=1452789005063, finishTime=1452789005272, timeTaken=209, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3, vertexStats=firstTaskStartTime=1452789005182, firstTasksToStart=[ task_1452789004804_0001_1_01_000000 ], lastTaskFinishTime=1452789005269, lastTasksToFinish=[ task_1452789004804_0001_1_01_000000 ], minTaskDuration=87, maxTaskDuration=87, avgTaskDuration=87.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_01_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,273 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,273 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] attempt: attempt_1452789004804_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,273 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,273 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,273 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,274 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,274 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000002, stoppedTime=1452789005274, exitStatus=0
2016-01-14 16:30:05,274 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,274 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,275 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskId=task_1452789004804_0001_1_02_000000, scheduledTime=1452789005275, launchTime=1452789005275
2016-01-14 16:30:05,275 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,275 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,275 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,275 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,275 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,276 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A016A996D30D455BA1D753AC00B9C434, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,276 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,276 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,276 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,277 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000003
2016-01-14 16:30:05,278 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,279 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789004804_0001_00_000003
2016-01-14 16:30:05,279 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:05,280 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,280 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,280 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000002, Exiting
2016-01-14 16:30:05,280 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,281 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,282 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,282 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000002 completed successfully
2016-01-14 16:30:05,284 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,284 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,284 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,285 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000003, launchTime=1452789005282
2016-01-14 16:30:05,285 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,286 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_02_000000_0] to container: [container_1452789004804_0001_00_000003]
2016-01-14 16:30:05,286 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,287 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_02_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,287 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, startTime=1452789005287, containerId=container_1452789004804_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:05,287 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,287 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,287 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,288 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000003 given task: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,289 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,289 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,289 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,289 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,289 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,289 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A016A996D30D455BA1D753AC00B9C434, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,289 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,290 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,290 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,291 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:05,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,294 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,297 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,297 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,298 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,299 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,299 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,299 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,300 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,300 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,307 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A016A996D30D455BA1D753AC00B9C434, ordinal: 2
2016-01-14 16:30:05,307 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A016A996D30D455BA1D753AC00B9C434, mem on start (mb), free: 84, total: 517, max: 672
2016-01-14 16:30:05,307 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,307 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,307 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,307 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,307 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,308 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,308 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,308 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,308 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:05,308 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D2A0861392DE4D708D767DEDBAB90939, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:05,308 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, AE7FFF9183E44A5CA31F76C68F5532F6, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:05,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D2A0861392DE4D708D767DEDBAB90939 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,310 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,311 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,311 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D2A0861392DE4D708D767DEDBAB90939 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,312 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:05,312 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,312 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,312 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,313 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,313 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,316 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_01_000000_0_10041, runDuration: 76697, ]
2016-01-14 16:30:05,319 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_01_000000_0_10041] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789005319, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:05,319 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,319 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:05,320 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D2A0861392DE4D708D767DEDBAB90939] #11 in 3ms
2016-01-14 16:30:05,320 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:05,320 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D2A0861392DE4D708D767DEDBAB90939] #11, status:false, isInterrupted:false
2016-01-14 16:30:05,321 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,321 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,321 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:05,321 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,321 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,321 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,322 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:05,322 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,322 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,322 INFO element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,323 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@59cf288; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@2f6c199; comparator=cascading.tuple.hadoop.util.TupleComparator@16202554; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,323 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,323 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,323 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,341 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:05,341 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:05,341 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:05,341 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:05,341 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:05,342 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] ADBFE874A1E34C0892F9528F790D6501, for 1 inputs
2016-01-14 16:30:05,342 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A016A996D30D455BA1D753AC00B9C434, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,343 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@405e2da9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6707d342
2016-01-14 16:30:05,364 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,364 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A016A996D30D455BA1D753AC00B9C434, mem on close (mb), free: 186, total: 503, max: 672
2016-01-14 16:30:05,364 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,365 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,366 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05,366 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,366 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,366 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 16:30:05,367 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:05,370 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,374 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,375 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_02, eventCount=1
2016-01-14 16:30:05,375 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 16:30:05,375 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,376 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:05,376 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,376 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,376 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, startTime=1452789005287, finishTime=1452789005375, timeTaken=88, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 16:30:05,376 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,377 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,377 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,377 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,378 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskId=task_1452789004804_0001_1_02_000000, startTime=1452789005287, finishTime=1452789005377, timeTaken=90, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 16:30:05,378 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,378 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] : 1
2016-01-14 16:30:05,378 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,379 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,380 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, vertexId=vertex_1452789004804_0001_1_02, initRequestedTime=1452789005036, initedTime=1452789005041, startRequestedTime=1452789005067, startedTime=1452789005067, finishTime=1452789005378, timeTaken=311, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452789005287, firstTasksToStart=[ task_1452789004804_0001_1_02_000000 ], lastTaskFinishTime=1452789005375, lastTasksToFinish=[ task_1452789004804_0001_1_02_000000 ], minTaskDuration=88, maxTaskDuration=88, avgTaskDuration=88.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_02_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,380 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,380 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,380 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000003, stoppedTime=1452789005380, exitStatus=0
2016-01-14 16:30:05,380 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,380 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,381 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] attempt: attempt_1452789004804_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,381 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,381 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,381 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,382 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,382 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskId=task_1452789004804_0001_1_03_000000, scheduledTime=1452789005382, launchTime=1452789005382
2016-01-14 16:30:05,382 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,382 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,383 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,383 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,383 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: AE7FFF9183E44A5CA31F76C68F5532F6, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:05,383 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,384 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,385 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000004
2016-01-14 16:30:05,385 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,386 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1452789004804_0001_00_000004
2016-01-14 16:30:05,387 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,387 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,387 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:05,387 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000003, Exiting
2016-01-14 16:30:05,388 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,388 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000004, launchTime=1452789005389
2016-01-14 16:30:05,391 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,391 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000003 completed successfully
2016-01-14 16:30:05,389 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,392 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,389 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,392 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,393 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_03_000000_0] to container: [container_1452789004804_0001_00_000004]
2016-01-14 16:30:05,393 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,394 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000004 given task: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,394 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,394 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_03_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,394 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,395 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,395 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,395 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,395 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,395 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,395 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,395 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: AE7FFF9183E44A5CA31F76C68F5532F6, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:05,396 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,395 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,395 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, startTime=1452789005394, containerId=container_1452789004804_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:05,396 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,397 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,398 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,396 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,399 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,401 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,401 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,402 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:05,402 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:05,403 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,403 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,403 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:05,403 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,405 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,405 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,405 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,405 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,409 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,417 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, ordinal: 3
2016-01-14 16:30:05,417 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, mem on start (mb), free: 179, total: 503, max: 672
2016-01-14 16:30:05,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,418 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,418 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,421 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:05,422 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,422 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,422 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,422 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:05,423 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:05,423 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,423 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,423 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:05,423 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B4B02E8B10CD404AAC3C9D5E7208B4A4, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:05,423 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A016A996D30D455BA1D753AC00B9C434, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:05,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A016A996D30D455BA1D753AC00B9C434 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,424 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,426 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,427 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A016A996D30D455BA1D753AC00B9C434 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,427 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:05,427 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,427 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,430 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,431 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_02_000000_0_10044, runDuration: 76941, ]
2016-01-14 16:30:05,433 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 16:30:05,434 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,434 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,434 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 16:30:05,435 INFO element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 6A1ACBEDEA3040F593F4450750E64F8E, for 1 inputs
2016-01-14 16:30:05,435 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_02_000000_0_10044] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452789005435, TimeTaken=1, Rate=4.89 MB/s
2016-01-14 16:30:05,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:05,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A016A996D30D455BA1D753AC00B9C434] #12 in 3ms
2016-01-14 16:30:05,438 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A016A996D30D455BA1D753AC00B9C434] #12, status:false, isInterrupted:false
2016-01-14 16:30:05,439 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,440 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 16:30:05,440 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,440 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,441 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 16:30:05,441 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,442 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,441 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, all 1 inputs ready in: 00:00:00.006
2016-01-14 16:30:05,445 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@244ff83; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7fe2f29e
2016-01-14 16:30:05,455 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:05,455 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:05,455 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,456 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789004804_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:05,457 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,457 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, mem on close (mb), free: 176, total: 503, max: 672
2016-01-14 16:30:05,458 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,459 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,459 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,459 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,460 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,460 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,462 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,464 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,464 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, startTime=1452789005394, finishTime=1452789005464, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:05,464 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,464 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,465 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,465 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskId=task_1452789004804_0001_1_03_000000, startTime=1452789005394, finishTime=1452789005464, timeTaken=70, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:05,465 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,465 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] : 1
2016-01-14 16:30:05,466 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,466 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,466 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, vertexId=vertex_1452789004804_0001_1_03, initRequestedTime=1452789005045, initedTime=1452789005059, startRequestedTime=1452789005070, startedTime=1452789005070, finishTime=1452789005466, timeTaken=396, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789005394, firstTasksToStart=[ task_1452789004804_0001_1_03_000000 ], lastTaskFinishTime=1452789005464, lastTasksToFinish=[ task_1452789004804_0001_1_03_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_03_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,467 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,467 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:05,467 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,468 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,468 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,468 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-01-14 16:30:05,468 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,469 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789004804_0001_1
2016-01-14 16:30:05,469 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: B4B02E8B10CD404AAC3C9D5E7208B4A4 for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,479 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,480 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,480 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,483 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789004804_0001_1, queueSize=0
2016-01-14 16:30:05,484 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:05,485 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789004804_0001_1, startTime=1452789005016, finishTime=1452789005480, timeTaken=464, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2140143616, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=30, MERGE_PHASE_TIME=45, FIRST_EVENT_RECEIVED=17, LAST_EVENT_RECEIVED=17, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156021160, Process_Duration=67, Process_End_Time=5811156021227, Read_Duration=19, Tuples_Read=303, Tuples_Written=317, Write_Duration=7, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:05,485 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789004804_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:05,485 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:05,486 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000004, stoppedTime=1452789005486, exitStatus=0
2016-01-14 16:30:05,487 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05 Completed Dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,489 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789004804_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:05,489 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789004804_0001_1_post
2016-01-14 16:30:05,490 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,490 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:05,491 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000004. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,491 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,494 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001
2016-01-14 16:30:05,496 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:05,496 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:05,496 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:05,496 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:05,496 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:05,497 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:05,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:05,498 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:05,498 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:05,498 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:05,498 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:05,499 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:05,499 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:05,499 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,499 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000004: 30 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,499 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000004, Exiting
2016-01-14 16:30:05,500 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,500 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,510 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001
2016-01-14 16:30:05,511 INFO flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] completed in: 00:00.726
2016-01-14 16:30:05,585 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short