Class cascading.DistanceUseCasePlatformTest

4

tests

0

failures

0

ignored

8.786s

duration

100%

successful

Tests

Test Duration Result
testEuclideanDistance 1.208s passed
testEuclideanDistanceComposite 5.041s passed
testEuclideanDistanceShort 1.106s passed
testPearsonDistanceComposite 1.431s passed

Standard output

2016-01-14 16:29:56,731 INFO  platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(195)) - found 1 test platforms from classpath
2016-01-14 16:29:56,765 INFO  platform.PlatformRunner (PlatformRunner.java:addPlatform(244)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-01-14 16:29:56,801 INFO  tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-01-14 16:29:57,185 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-14 16:29:57,549 INFO  platform.TestPlatform (TestPlatform.java:getGlobalProperties(73)) - platform property overrides: 
2016-01-14 16:29:57,805 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:29:57,806 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:29:57,919 INFO  property.AppProps (AppProps.java:getAppID(177)) - using app.id: E3C4D7DE265A4AC28A787666BB0CEA75
2016-01-14 16:29:58,199 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.153
2016-01-14 16:29:58,281 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.236
2016-01-14 16:29:58,282 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:29:58,283 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:29:58,284 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:29:58,425 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:29:58,632 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 16:29:58,906 INFO  util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.0.4
2016-01-14 16:29:58,908 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:29:58,908 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:29:58,909 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 16:29:58,909 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 16:29:58,909 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 16:29:58,910 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 16:29:58,911 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-01-14 16:29:58,911 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:29:58,938 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:29:59,134 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:29:59,140 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:29:59,140 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 16:29:59,182 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001 doesn't exist and is created
2016-01-14 16:29:59,240 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:29:59,241 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,242 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001
2016-01-14 16:29:59,270 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452788999139_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:29:59,310 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:29:59,324 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:29:59,341 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-01-14 16:29:59,341 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-01-14 16:29:59,341 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,348 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:29:59,352 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:29:59,355 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir
2016-01-14 16:29:59,355 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/history.txt.appattempt_1452788999139_0001_000000, maxErrors=10
2016-01-14 16:29:59,356 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:29:59,357 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452788999139_0001_000000, appSubmitTime=1452788999257, launchTime=1452788999269
2016-01-14 16:29:59,362 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:29:59,363 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:29:59,365 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:29:59,365 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:29:59,424 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:58623
2016-01-14 16:29:59,442 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-01-14 16:29:59,442 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-01-14 16:29:59,442 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:29:59,464 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:29:59,532 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452788999139_0001_000000, startTime=1452788999532
2016-01-14 16:29:59,533 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:29:59,539 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:29:59,542 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:29:59,560 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:29:59,560 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001, dagName=[BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:29:59,765 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:29:59,874 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452788999139_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/dag_1452788999139_0001_1.dot
2016-01-14 16:29:59,878 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001/localmode-log-dir/dag_1452788999139_0001_1-tez-dag.pb.txt
2016-01-14 16:30:00,041 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452788999139_0001_1
2016-01-14 16:30:00,042 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,042 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:30:00 Running Dag: dag_1452788999139_0001_1
2016-01-14 16:30:00,085 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:00,085 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452788999139_0001_1, submitTime=1452788999766
2016-01-14 16:30:00,116 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:00,123 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:00,126 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:00,126 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452788999139_0001_1, initTime=1452789000087
2016-01-14 16:30:00,128 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:00,130 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_STARTED]: dagID=dag_1452788999139_0001_1, startTime=1452789000129
2016-01-14 16:30:00,130 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:00,132 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:00,133 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:00,133 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: A7DD97C3B6B2413082CDEDDCDE1B7907 : {9298AB65CE5D47F4A732A1716A77AC67={InputName=9298AB65CE5D47F4A732A1716A77AC67}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:00,135 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 9298AB65CE5D47F4A732A1716A77AC67, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:00,135 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,142 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001, dagName=[BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite
2016-01-14 16:30:00,143 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,143 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,145 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]. Starting root input initializers: 1
2016-01-14 16:30:00,150 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 9298AB65CE5D47F4A732A1716A77AC67 on vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,164 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:00,165 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,177 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452788999139_0001, with dag id: dag_1452788999139_0001_1
2016-01-14 16:30:00,182 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:00,184 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 9298AB65CE5D47F4A732A1716A77AC67 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:00,185 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:00,186 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:00,196 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,196 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,233 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,238 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,239 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, vertexId=vertex_1452788999139_0001_1_01, initRequestedTime=1452789000164, initedTime=1452789000236, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,239 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,243 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,258 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:00,262 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 9298AB65CE5D47F4A732A1716A77AC67 on vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,269 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,269 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,270 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:00,270 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,271 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, vertexId=vertex_1452788999139_0001_1_02, initRequestedTime=1452789000243, initedTime=1452789000270, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,271 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,272 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:00,272 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {9298AB65CE5D47F4A732A1716A77AC67=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:00,272 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] parallelism set to 1
2016-01-14 16:30:00,273 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:00,282 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_00, eventCount=1
2016-01-14 16:30:00,287 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,287 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, vertexId=vertex_1452788999139_0001_1_00, initRequestedTime=1452789000133, initedTime=1452789000283, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:00,288 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] with distanceFromRoot: 0
2016-01-14 16:30:00,288 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:00,288 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,317 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:00,317 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,318 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:00,321 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,339 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=085EFCBD350A4927B7CA1ABBE36055AC, vertexName=E37A8363570048DA9BBF994AFA44BDD3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:00,353 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=085EFCBD350A4927B7CA1ABBE36055AC, vertexId=vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:00,357 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:00,357 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, vertexId=vertex_1452788999139_0001_1_03, initRequestedTime=1452789000288, initedTime=1452789000356, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:00,357 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:00,359 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:00,360 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,367 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,367 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_00, startRequestedTime=1452789000358, startedTime=1452789000358
2016-01-14 16:30:00,368 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,404 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:00,404 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskId=task_1452788999139_0001_1_00_000000, scheduledTime=1452789000401, launchTime=1452789000401
2016-01-14 16:30:00,404 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:00,405 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_00 for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,405 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] with distanceFromRoot: 1
2016-01-14 16:30:00,407 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,411 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A7DD97C3B6B2413082CDEDDCDE1B7907 in vertex: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,414 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DF8E8CD00C5149F3A7F2E38E12732E8D with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,414 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,414 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_01, startRequestedTime=1452789000407, startedTime=1452789000407
2016-01-14 16:30:00,415 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,417 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A7DD97C3B6B2413082CDEDDCDE1B7907, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,419 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:00,420 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_01 for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,420 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] with distanceFromRoot: 2
2016-01-14 16:30:00,420 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:00,420 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: DF8E8CD00C5149F3A7F2E38E12732E8D in vertex: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:00,421 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: EA7868BACC9E49EA8A22C7E7869174B8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,422 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,422 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_02, startRequestedTime=1452789000420, startedTime=1452789000420
2016-01-14 16:30:00,423 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,423 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452788999139_0001_1_02 for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] numStartedSources: 1 numSources: 1
2016-01-14 16:30:00,423 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] with distanceFromRoot: 3
2016-01-14 16:30:00,423 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: EA7868BACC9E49EA8A22C7E7869174B8 in vertex: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:00,424 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E37A8363570048DA9BBF994AFA44BDD3 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:00,424 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:00,425 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452788999139_0001_1_03, startRequestedTime=1452789000423, startedTime=1452789000423
2016-01-14 16:30:00,425 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:00,468 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:00,472 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:00,473 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:00,480 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000001
2016-01-14 16:30:00,480 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:00,488 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452788999139_0001_00_000001
2016-01-14 16:30:00,488 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:00,496 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:00,496 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:00,496 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000001, launchTime=1452789000496
2016-01-14 16:30:00,500 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,504 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_00_000000_0] to container: [container_1452788999139_0001_00_000001]
2016-01-14 16:30:00,506 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:00,514 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_00_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:00,514 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000001 given task: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,514 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,515 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,515 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,518 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:00,518 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:00,518 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:00,519 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:00,520 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:00,520 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, startTime=1452789000512, containerId=container_1452788999139_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:00,521 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:00,521 INFO  counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters:  GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-01-14 16:30:00,529 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:00,543 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:00,543 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: A7DD97C3B6B2413082CDEDDCDE1B7907, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,546 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:00,550 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,552 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:00,552 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,554 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:00,558 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=9298AB65CE5D47F4A732A1716A77AC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:00,559 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,559 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:00,561 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:00,561 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:00,563 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,567 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,568 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,576 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:00,578 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:00,578 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,587 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,588 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:00,590 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,590 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,608 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, ordinal: 0
2016-01-14 16:30:00,609 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, mem on start (mb), free: 218, total: 281, max: 672
2016-01-14 16:30:00,609 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,609 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:00,610 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:00,610 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:00,612 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:00,616 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:00,617 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:00,617 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:00,618 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:00,620 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:00,621 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 9298AB65CE5D47F4A732A1716A77AC67, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:00,621 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DF8E8CD00C5149F3A7F2E38E12732E8D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:00,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:00,623 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 9298AB65CE5D47F4A732A1716A77AC67 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:00,623 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,624 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:00,624 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,626 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:00,626 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:00,627 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:00,628 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,631 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:00,641 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@35097440
2016-01-14 16:30:00,653 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@35097440
2016-01-14 16:30:00,655 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:00,655 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:00,666 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:00,698 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 9298AB65CE5D47F4A732A1716A77AC67
2016-01-14 16:30:00,698 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:00,699 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:00,716 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@645abf3c; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@1c12f9d; comparator=cascading.tuple.hadoop.util.TupleComparator@321ced51; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:00,719 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:00,721 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:00,721 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:00,839 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:00,839 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:00,840 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:00,840 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:00,840 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:00,846 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:00,847 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:00,849 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, all 1 inputs ready in: 00:00:00.002
2016-01-14 16:30:00,860 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,860 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7DD97C3B6B2413082CDEDDCDE1B7907, mem on close (mb), free: 107, total: 281, max: 672
2016-01-14 16:30:00,860 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:00,861 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:00,861 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:00,861 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:00,879 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:00 Completed running task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,901 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:00,903 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_00, eventCount=1
2016-01-14 16:30:00,907 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:00,907 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:00,910 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskAttemptId=attempt_1452788999139_0001_1_00_000000_0, startTime=1452789000512, finishTime=1452789000903, timeTaken=391, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:00,912 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:00,913 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:00,916 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00,916 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:00,917 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,918 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,921 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:00,927 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, taskId=task_1452788999139_0001_1_00_000000, startTime=1452789000512, finishTime=1452789000920, timeTaken=408, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:00,929 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:00,930 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] : 1
2016-01-14 16:30:00,930 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:00,938 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:00,946 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, vertexId=vertex_1452788999139_0001_1_00, initRequestedTime=1452789000133, initedTime=1452789000283, startRequestedTime=1452789000358, startedTime=1452789000358, finishTime=1452789000931, timeTaken=573, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=2157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789000849, Process_Duration=11, Process_End_Time=1452789000860, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789000512, firstTasksToStart=[ task_1452788999139_0001_1_00_000000 ], lastTaskFinishTime=1452789000903, lastTasksToFinish=[ task_1452788999139_0001_1_00_000000 ], minTaskDuration=391, maxTaskDuration=391, avgTaskDuration=391.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_00_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:00,946 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:00,947 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:00,948 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:00,949 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:00,949 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:00,949 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000001, Exiting
2016-01-14 16:30:00,949 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,949 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000001
2016-01-14 16:30:00,953 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000001, stoppedTime=1452789000948, exitStatus=0
2016-01-14 16:30:00,954 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:00,954 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000001 completed successfully
2016-01-14 16:30:00,954 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:00,956 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] attempt: attempt_1452788999139_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:00,957 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:00,958 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,960 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:00,961 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452788999139_0001_00_000001
2016-01-14 16:30:00,961 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000001 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:00,962 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:00,962 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskId=task_1452788999139_0001_1_01_000000, scheduledTime=1452789000962, launchTime=1452789000962
2016-01-14 16:30:00,963 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:00,964 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:00,965 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: DF8E8CD00C5149F3A7F2E38E12732E8D, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,965 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:00,966 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:00,967 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000002
2016-01-14 16:30:00,968 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:00,971 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452788999139_0001_00_000002
2016-01-14 16:30:00,971 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:00,975 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:00,976 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000002, launchTime=1452789000975
2016-01-14 16:30:00,976 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:00,977 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000002
2016-01-14 16:30:00,979 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_01_000000_0] to container: [container_1452788999139_0001_00_000002]
2016-01-14 16:30:00,979 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:00,980 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000002 given task: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,980 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,981 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_01_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:00,981 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,982 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:00,982 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:00,982 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:00,982 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:00,983 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:00,983 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:00,984 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: DF8E8CD00C5149F3A7F2E38E12732E8D, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:00,984 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:00,986 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:00,986 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:00,986 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, startTime=1452789000980, containerId=container_1452788999139_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:00,988 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:00,988 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:00,989 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:00,988 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:00,990 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,992 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:00,992 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:00,993 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A7DD97C3B6B2413082CDEDDCDE1B7907, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:00,993 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:00,993 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:00,995 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:00,993 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,002 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:01,002 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,005 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,008 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,009 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,009 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,010 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,011 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,025 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, ordinal: 1
2016-01-14 16:30:01,025 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, mem on start (mb), free: 95, total: 281, max: 672
2016-01-14 16:30:01,025 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,025 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,025 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,025 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,026 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,026 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:01,026 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,027 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,027 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:01,027 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, EA7868BACC9E49EA8A22C7E7869174B8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:01,027 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A7DD97C3B6B2413082CDEDDCDE1B7907, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:01,027 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,027 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A7DD97C3B6B2413082CDEDDCDE1B7907 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,027 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,028 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,028 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,038 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,040 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,044 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A7DD97C3B6B2413082CDEDDCDE1B7907 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,046 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:01,046 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,051 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,054 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,054 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,054 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,058 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 16:30:01,064 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_00_000000_0_10002, runDuration: 315011, ]
2016-01-14 16:30:01,064 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 790B4DDE3C8A434C8B2994E08F37E4F6
2016-01-14 16:30:01,066 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,066 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,068 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@55c00779; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@322ac4ea; comparator=cascading.tuple.hadoop.util.TupleComparator@15e5f861; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:01,069 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:01,069 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:01,070 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,072 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_00_000000_0_10002] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789001072, TimeTaken=3, Rate=0.35 MB/s
2016-01-14 16:30:01,139 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,138 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:01,140 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:01,140 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:01,140 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:01,140 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:01,141 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A7DD97C3B6B2413082CDEDDCDE1B7907] #1 in 72ms
2016-01-14 16:30:01,141 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:01,142 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A7DD97C3B6B2413082CDEDDCDE1B7907] #1, status:false, isInterrupted:false
2016-01-14 16:30:01,143 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,144 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 790B4DDE3C8A434C8B2994E08F37E4F6, for 1 inputs
2016-01-14 16:30:01,146 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:01,146 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,150 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,151 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:01,151 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:01,154 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:01,154 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,155 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,158 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:01,161 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@325c4a21; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4906d65b
2016-01-14 16:30:01,212 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,212 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF8E8CD00C5149F3A7F2E38E12732E8D, mem on close (mb), free: 87, total: 345, max: 672
2016-01-14 16:30:01,212 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A7DD97C3B6B2413082CDEDDCDE1B7907
2016-01-14 16:30:01,213 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,213 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,213 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,214 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,214 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01,215 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:01,215 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:01,215 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:01,216 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:01,252 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,257 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,258 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_01, eventCount=1
2016-01-14 16:30:01,258 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,262 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13]]
2016-01-14 16:30:01,263 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:01,263 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,263 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,265 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskAttemptId=attempt_1452788999139_0001_1_01_000000_0, startTime=1452789000980, finishTime=1452789001258, timeTaken=278, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13
2016-01-14 16:30:01,266 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,267 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01,267 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,268 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,270 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, taskId=task_1452788999139_0001_1_01_000000, startTime=1452789000980, finishTime=1452789001267, timeTaken=287, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13
2016-01-14 16:30:01,271 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,271 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] : 1
2016-01-14 16:30:01,271 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:01,273 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,279 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, vertexId=vertex_1452788999139_0001_1_01, initRequestedTime=1452789000164, initedTime=1452789000236, startRequestedTime=1452789000407, startedTime=1452789000407, finishTime=1452789001272, timeTaken=865, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=361758720, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=102, MERGE_PHASE_TIME=114, FIRST_EVENT_RECEIVED=30, LAST_EVENT_RECEIVED=30, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001158, Process_Duration=54, Process_End_Time=1452789001212, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=13, vertexStats=firstTaskStartTime=1452789000980, firstTasksToStart=[ task_1452788999139_0001_1_01_000000 ], lastTaskFinishTime=1452789001258, lastTasksToFinish=[ task_1452788999139_0001_1_01_000000 ], minTaskDuration=278, maxTaskDuration=278, avgTaskDuration=278.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_01_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,280 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,280 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,281 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000002, stoppedTime=1452789001280, exitStatus=0
2016-01-14 16:30:01,281 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,282 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] attempt: attempt_1452788999139_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:01,281 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,283 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,284 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,284 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,285 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452788999139_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:01,286 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:01,287 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskId=task_1452788999139_0001_1_02_000000, scheduledTime=1452789001286, launchTime=1452789001286
2016-01-14 16:30:01,287 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:01,287 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:01,288 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,288 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EA7868BACC9E49EA8A22C7E7869174B8, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:01,289 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:01,289 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:01,290 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000003
2016-01-14 16:30:01,291 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:01,292 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452788999139_0001_00_000003
2016-01-14 16:30:01,292 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:01,293 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:01,293 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:01,295 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000003, launchTime=1452789001293
2016-01-14 16:30:01,297 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,298 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,299 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000002: 32 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,299 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000002, Exiting
2016-01-14 16:30:01,299 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,299 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000002
2016-01-14 16:30:01,302 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,302 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_02_000000_0] to container: [container_1452788999139_0001_00_000003]
2016-01-14 16:30:01,302 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000002 completed successfully
2016-01-14 16:30:01,303 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:01,303 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01,303 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000003 given task: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,304 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,304 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_02_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:01,305 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:01,305 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, startTime=1452789001304, containerId=container_1452788999139_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:01,305 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:01,306 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,307 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,308 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,309 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:01,309 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:01,310 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:01,310 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:01,311 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:01,311 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EA7868BACC9E49EA8A22C7E7869174B8, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:01,312 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:01,313 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,314 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:01,314 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,315 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:01,316 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DF8E8CD00C5149F3A7F2E38E12732E8D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:01,316 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E37A8363570048DA9BBF994AFA44BDD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:01,316 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,318 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:01,316 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:01,318 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,318 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,321 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:01,321 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,321 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,322 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,322 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,322 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,325 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,326 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,337 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, ordinal: 2
2016-01-14 16:30:01,337 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, mem on start (mb), free: 80, total: 345, max: 672
2016-01-14 16:30:01,337 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,338 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,338 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,339 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,339 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,339 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:01,340 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,340 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,341 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:01,341 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E37A8363570048DA9BBF994AFA44BDD3, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:01,341 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DF8E8CD00C5149F3A7F2E38E12732E8D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:01,342 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,342 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DF8E8CD00C5149F3A7F2E38E12732E8D being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,342 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,343 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,342 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,346 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,347 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,347 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: DF8E8CD00C5149F3A7F2E38E12732E8D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,348 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:01,348 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,349 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,351 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,351 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,353 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,353 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_01_000000_0_10005, runDuration: 256187, ]
2016-01-14 16:30:01,357 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_01_000000_0_10005] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789001356, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:01,358 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,358 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:01,358 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [DF8E8CD00C5149F3A7F2E38E12732E8D] #2 in 3ms
2016-01-14 16:30:01,360 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [DF8E8CD00C5149F3A7F2E38E12732E8D] #2, status:false, isInterrupted:false
2016-01-14 16:30:01,361 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:01,361 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,361 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:01,362 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,362 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,363 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:01,363 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,364 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,365 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 1C61807F015844DEB80C5A0FA6B960BE
2016-01-14 16:30:01,365 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,366 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,367 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@70418842; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@5e58dea1; comparator=cascading.tuple.hadoop.util.TupleComparator@1c804c6b; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:01,367 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:01,368 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:01,368 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,437 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:01,438 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:01,438 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:01,438 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:01,438 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:01,439 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 1C61807F015844DEB80C5A0FA6B960BE, for 1 inputs
2016-01-14 16:30:01,439 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:01,440 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@58416461; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6aff5bd4
2016-01-14 16:30:01,467 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,468 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EA7868BACC9E49EA8A22C7E7869174B8, mem on close (mb), free: 76, total: 408, max: 672
2016-01-14 16:30:01,468 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: DF8E8CD00C5149F3A7F2E38E12732E8D
2016-01-14 16:30:01,469 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,469 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,470 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,470 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,470 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01,471 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:01,471 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:01,472 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 16:30:01,472 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:01,481 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,487 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,488 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452788999139_0001_1_02, eventCount=1
2016-01-14 16:30:01,489 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,491 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskAttemptId=attempt_1452788999139_0001_1_02_000000_0, startTime=1452789001304, finishTime=1452789001488, timeTaken=184, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 16:30:01,492 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,492 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01,492 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,493 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,494 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, taskId=task_1452788999139_0001_1_02_000000, startTime=1452789001304, finishTime=1452789001492, timeTaken=188, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6
2016-01-14 16:30:01,494 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,494 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] : 1
2016-01-14 16:30:01,494 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,497 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6]]
2016-01-14 16:30:01,498 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:01,498 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,497 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=EA7868BACC9E49EA8A22C7E7869174B8, vertexId=vertex_1452788999139_0001_1_02, initRequestedTime=1452789000243, initedTime=1452789000270, startRequestedTime=1452789000420, startedTime=1452789000420, finishTime=1452789001494, timeTaken=1074, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001440, Process_Duration=27, Process_End_Time=1452789001467, Read_Duration=14, Tuples_Read=174, Tuples_Written=87, Write_Duration=6, vertexStats=firstTaskStartTime=1452789001304, firstTasksToStart=[ task_1452788999139_0001_1_02_000000 ], lastTaskFinishTime=1452789001488, lastTasksToFinish=[ task_1452788999139_0001_1_02_000000 ], minTaskDuration=184, maxTaskDuration=184, avgTaskDuration=184.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_02_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,500 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,497 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,500 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,501 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,501 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000003, stoppedTime=1452789001501, exitStatus=0
2016-01-14 16:30:01,501 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,501 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] attempt: attempt_1452788999139_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:01,501 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E37A8363570048DA9BBF994AFA44BDD3
2016-01-14 16:30:01,502 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,502 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,502 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,502 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452788999139_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:01,502 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,503 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,503 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000003, Exiting
2016-01-14 16:30:01,503 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,503 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_STARTED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskId=task_1452788999139_0001_1_03_000000, scheduledTime=1452789001503, launchTime=1452789001503
2016-01-14 16:30:01,503 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000003
2016-01-14 16:30:01,503 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:01,504 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,504 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:01,505 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000003 completed successfully
2016-01-14 16:30:01,507 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:01,507 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,512 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01,512 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E37A8363570048DA9BBF994AFA44BDD3, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:01,512 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:01,512 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:01,513 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452788999139_0001_00_000004
2016-01-14 16:30:01,513 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:01,515 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452788999139_0001_00_000004
2016-01-14 16:30:01,516 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:01,518 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:01,518 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452788999139_0001_00_000004, launchTime=1452789001518
2016-01-14 16:30:01,518 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:01,519 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,520 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452788999139_0001_1_03_000000_0] to container: [container_1452788999139_0001_00_000004]
2016-01-14 16:30:01,520 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:01,520 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452788999139_0001_00_000004 given task: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,520 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,521 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452788999139_0001_1_03_000000_0] started. Is using containerId: [container_1452788999139_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:01,521 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,521 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,521 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:01,522 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:01,522 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:01,522 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:01,522 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:01,522 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788] euclidean/(1/1) ...osite/euclidean/composite, VertexName: E37A8363570048DA9BBF994AFA44BDD3, VertexParallelism: 1, TaskAttemptID:attempt_1452788999139_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:01,521 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, startTime=1452789001520, containerId=container_1452788999139_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452788999139_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:01,523 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:01,524 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:01,521 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:01,522 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:01,527 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,529 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:01,529 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,528 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452788999139_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:01,530 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=EA7868BACC9E49EA8A22C7E7869174B8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:01,530 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=085EFCBD350A4927B7CA1ABBE36055AC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:01,530 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,531 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:01,532 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,534 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:01,534 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:01,535 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,536 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,537 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:01,537 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:01,538 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,538 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:01,538 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,561 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, ordinal: 3
2016-01-14 16:30:01,561 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, mem on start (mb), free: 69, total: 408, max: 672
2016-01-14 16:30:01,562 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:01,562 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:01,562 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:01,570 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:01,570 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,570 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:01,571 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:01,571 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:01,571 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:01,571 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:01,571 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:01,572 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:01,572 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, EA7868BACC9E49EA8A22C7E7869174B8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:01,572 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 085EFCBD350A4927B7CA1ABBE36055AC, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:01,572 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:01,572 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: EA7868BACC9E49EA8A22C7E7869174B8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:01,573 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:01,573 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:01,573 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,575 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:01,575 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:01,575 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: EA7868BACC9E49EA8A22C7E7869174B8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:01,576 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:01,576 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:01,576 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:01,577 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,579 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:01,580 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,583 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452788999139_0001_1_02_000000_0_10009, runDuration: 166032, ]
2016-01-14 16:30:01,586 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-01-14 16:30:01,588 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452788999139_0001_1_02_000000_0_10009] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452789001588, TimeTaken=3, Rate=1.38 MB/s
2016-01-14 16:30:01,588 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,589 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: 47AC7260A856421A87B882BB7B76D34B
2016-01-14 16:30:01,589 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:01,589 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: 085EFCBD350A4927B7CA1ABBE36055AC
2016-01-14 16:30:01,590 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-01-14 16:30:01,589 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [EA7868BACC9E49EA8A22C7E7869174B8] #3 in 4ms
2016-01-14 16:30:01,591 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [EA7868BACC9E49EA8A22C7E7869174B8] #3, status:false, isInterrupted:false
2016-01-14 16:30:01,591 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] 47AC7260A856421A87B882BB7B76D34B, for 1 inputs
2016-01-14 16:30:01,592 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:01,593 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 16:30:01,593 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,593 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:01,594 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 16:30:01,594 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,594 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:01,595 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, all 1 inputs ready in: 00:00:00.003
2016-01-14 16:30:01,596 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1a08caf5; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3744a82d
2016-01-14 16:30:01,612 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:01,612 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:01,613 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,613 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452788999139_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:01,614 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,614 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E37A8363570048DA9BBF994AFA44BDD3, mem on close (mb), free: 65, total: 408, max: 672
2016-01-14 16:30:01,615 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: EA7868BACC9E49EA8A22C7E7869174B8
2016-01-14 16:30:01,615 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:01,616 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:01,616 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:01,616 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:01,616 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,617 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:01,619 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:01,619 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:01,620 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskAttemptId=attempt_1452788999139_0001_1_03_000000_0, startTime=1452789001520, finishTime=1452789001618, timeTaken=98, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:01,621 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452788999139_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:01,621 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:01,621 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:01,622 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,621 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01,622 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,623 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:01,625 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:TASK_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, taskId=task_1452788999139_0001_1_03_000000, startTime=1452789001520, finishTime=1452789001622, timeTaken=102, status=SUCCEEDED, successfulAttemptID=attempt_1452788999139_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:01,625 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452788999139_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:01,626 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] : 1
2016-01-14 16:30:01,626 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,627 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:01,628 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:VERTEX_FINISHED]: vertexName=E37A8363570048DA9BBF994AFA44BDD3, vertexId=vertex_1452788999139_0001_1_03, initRequestedTime=1452789000288, initedTime=1452789000356, startRequestedTime=1452789000423, startedTime=1452789000423, finishTime=1452789001626, timeTaken=1203, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=428343296, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789001595, Process_Duration=19, Process_End_Time=1452789001614, Read_Duration=2, Tuples_Read=87, Tuples_Written=21, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789001520, firstTasksToStart=[ task_1452788999139_0001_1_03_000000 ], lastTaskFinishTime=1452789001618, lastTasksToFinish=[ task_1452788999139_0001_1_03_000000 ], minTaskDuration=98, maxTaskDuration=98, avgTaskDuration=98.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452788999139_0001_1_03_000000 ], longestDurationTasks=[ task_1452788999139_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:01,629 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:01,629 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:01,630 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452788999139_0001_00_000004, stoppedTime=1452789001630, exitStatus=0
2016-01-14 16:30:01,630 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:01,631 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:01,631 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452788999139_0001_1
2016-01-14 16:30:01,631 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:01,633 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452788999139_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:01,633 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452788999139_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:01,633 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452788999139_0001_00_000004, Exiting
2016-01-14 16:30:01,633 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,633 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452788999139_0001_00_000004
2016-01-14 16:30:01,635 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452788999139_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:01,635 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452788999139_0001_00_000004 completed successfully
2016-01-14 16:30:01,636 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452788999139_0001_1
2016-01-14 16:30:01,638 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_00 [A7DD97C3B6B2413082CDEDDCDE1B7907]
2016-01-14 16:30:01,639 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_01 [DF8E8CD00C5149F3A7F2E38E12732E8D]
2016-01-14 16:30:01,639 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452788999139_0001_1_02 [EA7868BACC9E49EA8A22C7E7869174B8]
2016-01-14 16:30:01,640 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 085EFCBD350A4927B7CA1ABBE36055AC for vertex: vertex_1452788999139_0001_1_03 [E37A8363570048DA9BBF994AFA44BDD3]
2016-01-14 16:30:01,662 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452788999139_0001_1, queueSize=0
2016-01-14 16:30:01,662 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:01,663 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452788999139_0001_1][Event:DAG_FINISHED]: dagId=dag_1452788999139_0001_1, startTime=1452789000129, finishTime=1452789001653, timeTaken=1524, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=16024, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1513619456, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=130, MERGE_PHASE_TIME=149, FIRST_EVENT_RECEIVED=47, LAST_EVENT_RECEIVED=47, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156005042, Process_Duration=111, Process_End_Time=5811156005153, Read_Duration=21, Tuples_Read=303, Tuples_Written=317, Write_Duration=22, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:01,665 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452788999139_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:01,665 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452788999139_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:01,665 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452788999139_0001_00_000004
2016-01-14 16:30:01,666 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452788999139_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:01 Completed Dag: dag_1452788999139_0001_1
2016-01-14 16:30:01,667 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452788999139_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:01,667 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452788999139_0001_1_post
2016-01-14 16:30:01,667 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:01,668 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:01,679 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1452788999139_0001
2016-01-14 16:30:01,682 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:01,683 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:01,683 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:01,683 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:01,683 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:01,684 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:01,687 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:01,687 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:01,687 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:01,687 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:01,688 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:01,688 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:01,688 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:01,696 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/BF6F088C380642299EDDE702E521E736/ACB97D2F28764663B51A3998BA67E788/.tez/application_1452788999139_0001
2016-01-14 16:30:01,697 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:02.788
2016-01-14 16:30:01,838 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-01-14 16:30:01,876 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:01,876 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:01,982 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.072
2016-01-14 16:30:02,009 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.099
2016-01-14 16:30:02,010 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:02,011 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:02,011 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:02,037 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:02,073 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 16:30:02,258 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting
2016-01-14 16:30:02,259 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,259 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 16:30:02,260 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  parallel execution of steps is enabled: false
2016-01-14 16:30:02,261 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  executing total steps: 1
2016-01-14 16:30:02,261 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  allocating management threads: 1
2016-01-14 16:30:02,263 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,263 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:02,282 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] tez session mode enabled: true
2016-01-14 16:30:02,288 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:02,288 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:02,288 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:02,300 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001 doesn't exist and is created
2016-01-14 16:30:02,323 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:02,323 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:02,324 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001
2016-01-14 16:30:02,329 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789002288_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:02,343 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:02,344 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:02,346 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:02,346 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:02,346 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir
2016-01-14 16:30:02,346 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/history.txt.appattempt_1452789002288_0001_000000, maxErrors=10
2016-01-14 16:30:02,347 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:02,347 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789002288_0001_000000, appSubmitTime=1452789002329, launchTime=1452789002329
2016-01-14 16:30:02,351 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:02,354 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:02,355 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:36788
2016-01-14 16:30:02,360 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:02,360 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:02,362 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:02,364 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:02,364 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789002288_0001_000000, startTime=1452789002364
2016-01-14 16:30:02,364 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:02,424 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:02,424 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:02,424 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001, dagName=[85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,449 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,453 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789002288_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/dag_1452789002288_0001_1.dot
2016-01-14 16:30:02,455 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001/localmode-log-dir/dag_1452789002288_0001_1-tez-dag.pb.txt
2016-01-14 16:30:02,501 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789002288_0001_1
2016-01-14 16:30:02,501 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,501 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02 Running Dag: dag_1452789002288_0001_1
2016-01-14 16:30:02,521 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:02,521 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789002288_0001_1, submitTime=1452789002449
2016-01-14 16:30:02,523 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,523 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:02,523 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:02,524 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789002288_0001_1, initTime=1452789002521
2016-01-14 16:30:02,524 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:02,525 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_STARTED]: dagID=dag_1452789002288_0001_1, startTime=1452789002524
2016-01-14 16:30:02,525 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:02,525 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:02,525 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:02,525 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: A70FD1D546294A2AB0C612D3B44665F7 : {C900CBB86FF24138ADC3D29199F1C209={InputName=C900CBB86FF24138ADC3D29199F1C209}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:02,525 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: C900CBB86FF24138ADC3D29199F1C209, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:02,526 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001, dagName=[85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite
2016-01-14 16:30:02,527 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,527 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,528 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,528 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]. Starting root input initializers: 1
2016-01-14 16:30:02,528 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:02,528 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: C900CBB86FF24138ADC3D29199F1C209 on vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,529 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,532 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:02,532 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input C900CBB86FF24138ADC3D29199F1C209 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:02,534 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:02,534 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:02,534 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,535 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,535 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,536 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,536 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, vertexId=vertex_1452789002288_0001_1_01, initRequestedTime=1452789002529, initedTime=1452789002535, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,536 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,536 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,538 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:02,539 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: C900CBB86FF24138ADC3D29199F1C209 on vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,542 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,542 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,542 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,542 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,542 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, vertexId=vertex_1452789002288_0001_1_02, initRequestedTime=1452789002536, initedTime=1452789002542, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,543 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,543 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:02,543 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {C900CBB86FF24138ADC3D29199F1C209=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:02,543 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] parallelism set to 1
2016-01-14 16:30:02,544 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:02,545 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_00, eventCount=1
2016-01-14 16:30:02,547 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, vertexId=vertex_1452789002288_0001_1_00, initRequestedTime=1452789002525, initedTime=1452789002545, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:02,547 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,547 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] with distanceFromRoot: 0
2016-01-14 16:30:02,548 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:02,548 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,551 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson] submitted tez dag to app master: application_1452789002288_0001, with dag id: dag_1452789002288_0001_1
2016-01-14 16:30:02,577 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:02,577 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,578 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:02,578 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,583 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=668F7BA666CF49C2884A386FFBEAC8AF, vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:02,583 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=668F7BA666CF49C2884A386FFBEAC8AF, vertexId=vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:02,585 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, vertexId=vertex_1452789002288_0001_1_03, initRequestedTime=1452789002548, initedTime=1452789002585, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:02,585 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:02,585 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:02,585 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,586 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,586 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,586 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_00, startRequestedTime=1452789002585, startedTime=1452789002585
2016-01-14 16:30:02,587 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,589 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,589 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskId=task_1452789002288_0001_1_00_000000, scheduledTime=1452789002588, launchTime=1452789002588
2016-01-14 16:30:02,590 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,590 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_00 for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,591 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] with distanceFromRoot: 1
2016-01-14 16:30:02,592 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,593 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A70FD1D546294A2AB0C612D3B44665F7 in vertex: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,593 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C6F906C21EA04AB8AEC5AF372B2F50C8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,593 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,594 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_01, startRequestedTime=1452789002593, startedTime=1452789002593
2016-01-14 16:30:02,594 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,595 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: A70FD1D546294A2AB0C612D3B44665F7, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,595 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,595 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_01 for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,596 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] with distanceFromRoot: 2
2016-01-14 16:30:02,596 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,596 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C6F906C21EA04AB8AEC5AF372B2F50C8 in vertex: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,597 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: ECA94FF0DA97475897C9BC9E720E2D3C with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,597 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:02,597 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_02, startRequestedTime=1452789002596, startedTime=1452789002596
2016-01-14 16:30:02,597 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,598 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,599 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:02,599 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:02,600 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000001
2016-01-14 16:30:02,600 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,601 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789002288_0001_1_02 for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] numStartedSources: 1 numSources: 1
2016-01-14 16:30:02,601 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] with distanceFromRoot: 3
2016-01-14 16:30:02,603 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: ECA94FF0DA97475897C9BC9E720E2D3C in vertex: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,604 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DDAF1DA7D9CA4B3EBC36522E39432376 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:02,604 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:02,604 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789002288_0001_1_03, startRequestedTime=1452789002603, startedTime=1452789002603
2016-01-14 16:30:02,604 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:02,604 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789002288_0001_00_000001
2016-01-14 16:30:02,604 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:02,607 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,607 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000001, launchTime=1452789002606
2016-01-14 16:30:02,607 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,608 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,609 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_00_000000_0] to container: [container_1452789002288_0001_00_000001]
2016-01-14 16:30:02,609 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,609 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000001 given task: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,609 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,609 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_00_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,610 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, startTime=1452789002609, containerId=container_1452789002288_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:02,610 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,611 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,611 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,615 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,615 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,615 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,615 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,615 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,616 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,616 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,616 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: A70FD1D546294A2AB0C612D3B44665F7, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,616 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,619 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:02,621 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,621 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,621 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C900CBB86FF24138ADC3D29199F1C209, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:02,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,622 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,622 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,624 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,624 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,627 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,627 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,627 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:02,628 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,628 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,627 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,629 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,643 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, ordinal: 0
2016-01-14 16:30:02,643 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, mem on start (mb), free: 254, total: 516, max: 672
2016-01-14 16:30:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,644 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,644 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,644 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:02,645 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:02,645 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,645 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,645 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:02,645 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C900CBB86FF24138ADC3D29199F1C209, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:02,646 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C6F906C21EA04AB8AEC5AF372B2F50C8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C900CBB86FF24138ADC3D29199F1C209 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,646 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:02,647 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,646 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,647 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,648 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,649 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,649 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:02,650 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@432659c7
2016-01-14 16:30:02,652 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@432659c7
2016-01-14 16:30:02,652 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:02,652 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:02,655 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,656 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: C900CBB86FF24138ADC3D29199F1C209
2016-01-14 16:30:02,657 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,657 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,658 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@49e1985a; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@79d9629a; comparator=cascading.tuple.hadoop.util.TupleComparator@2ed8aaf5; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,659 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,659 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,659 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,676 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:02,677 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:02,677 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:02,678 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:02,678 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:02,680 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:02,680 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:02,681 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:02,686 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,686 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A70FD1D546294A2AB0C612D3B44665F7, mem on close (mb), free: 151, total: 516, max: 672
2016-01-14 16:30:02,686 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:02,687 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:02,687 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:02,687 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:02,692 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,697 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:02,698 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_00, eventCount=1
2016-01-14 16:30:02,698 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:02,699 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:02,699 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:02,700 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,701 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,701 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskAttemptId=attempt_1452789002288_0001_1_00_000000_0, startTime=1452789002609, finishTime=1452789002699, timeTaken=90, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:02,702 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:02,702 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:02,702 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02,703 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:02,703 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, taskId=task_1452789002288_0001_1_00_000000, startTime=1452789002609, finishTime=1452789002703, timeTaken=94, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:02,704 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:02,704 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] : 1
2016-01-14 16:30:02,704 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:02,706 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:02,707 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=A70FD1D546294A2AB0C612D3B44665F7, vertexId=vertex_1452789002288_0001_1_00, initRequestedTime=1452789002525, initedTime=1452789002545, startRequestedTime=1452789002585, startedTime=1452789002585, finishTime=1452789002704, timeTaken=119, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002681, Process_Duration=5, Process_End_Time=1452789002686, Read_Duration=3, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789002609, firstTasksToStart=[ task_1452789002288_0001_1_00_000000 ], lastTaskFinishTime=1452789002699, lastTasksToFinish=[ task_1452789002288_0001_1_00_000000 ], minTaskDuration=90, maxTaskDuration=90, avgTaskDuration=90.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_00_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:02,708 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:02,708 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:02,709 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000001, stoppedTime=1452789002709, exitStatus=0
2016-01-14 16:30:02,709 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:02,709 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:02,710 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] attempt: attempt_1452789002288_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:02,710 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,710 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,711 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:02,711 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:02,711 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,711 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskId=task_1452789002288_0001_1_01_000000, scheduledTime=1452789002711, launchTime=1452789002711
2016-01-14 16:30:02,712 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,712 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:02,712 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,712 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:02,713 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: C6F906C21EA04AB8AEC5AF372B2F50C8, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,713 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,713 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,713 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:02,715 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000001, Exiting
2016-01-14 16:30:02,715 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,716 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000002
2016-01-14 16:30:02,716 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000001
2016-01-14 16:30:02,718 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,719 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:02,721 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789002288_0001_00_000002
2016-01-14 16:30:02,722 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000001 completed successfully
2016-01-14 16:30:02,723 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:02,723 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:02,726 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,726 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,726 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000002, launchTime=1452789002726
2016-01-14 16:30:02,727 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,729 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_01_000000_0] to container: [container_1452789002288_0001_00_000002]
2016-01-14 16:30:02,730 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,730 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_01_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,731 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, startTime=1452789002730, containerId=container_1452789002288_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:02,731 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,731 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02,731 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000002 given task: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,732 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,733 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,733 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,733 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,733 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,733 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,733 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: C6F906C21EA04AB8AEC5AF372B2F50C8, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,734 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,734 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,737 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,738 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,738 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,739 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A70FD1D546294A2AB0C612D3B44665F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:02,740 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,741 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,740 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,742 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,744 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:02,746 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,746 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,748 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,749 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:02,749 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:02,749 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,750 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,751 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,751 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,752 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,761 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, ordinal: 1
2016-01-14 16:30:02,761 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, mem on start (mb), free: 144, total: 516, max: 672
2016-01-14 16:30:02,761 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,762 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,762 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,762 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,762 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,763 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:02,763 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,763 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,764 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:02,764 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A70FD1D546294A2AB0C612D3B44665F7, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:02,764 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, ECA94FF0DA97475897C9BC9E720E2D3C, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:02,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A70FD1D546294A2AB0C612D3B44665F7 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,765 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,766 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,767 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,767 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:02,767 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A70FD1D546294A2AB0C612D3B44665F7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:02,768 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:02,768 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:02,768 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:02,769 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,769 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,770 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,771 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_00_000000_0_10015, runDuration: 71633, ]
2016-01-14 16:30:02,772 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_00_000000_0_10015] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789002772, TimeTaken=1, Rate=1.04 MB/s
2016-01-14 16:30:02,772 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,773 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:02,773 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A70FD1D546294A2AB0C612D3B44665F7] #4 in 2ms
2016-01-14 16:30:02,773 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A70FD1D546294A2AB0C612D3B44665F7] #4, status:false, isInterrupted:false
2016-01-14 16:30:02,774 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:02,775 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:02,775 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,775 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:02,776 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:02,776 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,776 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:02,780 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-01-14 16:30:02,780 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: E45EB07641724802B9BBF837A8D6493F
2016-01-14 16:30:02,780 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,781 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,782 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@40403a1e; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@38a7a886; comparator=cascading.tuple.hadoop.util.TupleComparator@7291279; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,783 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,783 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,783 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,851 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:02,851 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:02,851 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:02,851 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:02,852 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:02,852 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] E45EB07641724802B9BBF837A8D6493F, for 1 inputs
2016-01-14 16:30:02,853 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:02,853 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:02,854 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:02,854 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@56f35464; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1299dd5
2016-01-14 16:30:02,883 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,883 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6F906C21EA04AB8AEC5AF372B2F50C8, mem on close (mb), free: 75, total: 516, max: 672
2016-01-14 16:30:02,884 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A70FD1D546294A2AB0C612D3B44665F7
2016-01-14 16:30:02,884 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:02,884 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:02,884 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,884 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:02,884 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:02,885 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:02,885 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:02,885 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:02,885 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:02,905 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,909 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:02,910 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_01, eventCount=1
2016-01-14 16:30:02,910 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5]]
2016-01-14 16:30:02,911 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:02,911 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:02,912 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskAttemptId=attempt_1452789002288_0001_1_01_000000_0, startTime=1452789002730, finishTime=1452789002911, timeTaken=181, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5
2016-01-14 16:30:02,912 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:02,912 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02,913 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,912 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:02,914 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,914 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:02,914 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, taskId=task_1452789002288_0001_1_01_000000, startTime=1452789002730, finishTime=1452789002912, timeTaken=182, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5
2016-01-14 16:30:02,915 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:02,915 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] : 1
2016-01-14 16:30:02,915 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:02,916 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:02,917 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, vertexId=vertex_1452789002288_0001_1_01, initRequestedTime=1452789002529, initedTime=1452789002535, startRequestedTime=1452789002593, startedTime=1452789002593, finishTime=1452789002916, timeTaken=323, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=8071, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789002854, Process_Duration=29, Process_End_Time=1452789002883, Read_Duration=0, Tuples_Read=35, Tuples_Written=174, Write_Duration=5, vertexStats=firstTaskStartTime=1452789002730, firstTasksToStart=[ task_1452789002288_0001_1_01_000000 ], lastTaskFinishTime=1452789002911, lastTasksToFinish=[ task_1452789002288_0001_1_01_000000 ], minTaskDuration=181, maxTaskDuration=181, avgTaskDuration=181.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_01_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:02,917 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:02,918 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:02,918 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000002, stoppedTime=1452789002918, exitStatus=0
2016-01-14 16:30:02,918 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:02,918 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] attempt: attempt_1452789002288_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:02,919 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:02,919 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,919 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:02,918 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:02,919 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:02,920 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskId=task_1452789002288_0001_1_02_000000, scheduledTime=1452789002920, launchTime=1452789002920
2016-01-14 16:30:02,920 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:02,921 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:02,921 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:02,921 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:02,921 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: ECA94FF0DA97475897C9BC9E720E2D3C, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,921 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:02,922 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:02,922 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000003
2016-01-14 16:30:02,923 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:02,924 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789002288_0001_00_000003
2016-01-14 16:30:02,924 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:02,925 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:02,926 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:02,926 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:02,926 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000002, Exiting
2016-01-14 16:30:02,926 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,926 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:02,926 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000002
2016-01-14 16:30:02,928 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000003, launchTime=1452789002925
2016-01-14 16:30:02,928 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:02,928 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000002 completed successfully
2016-01-14 16:30:02,929 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000003
2016-01-14 16:30:02,929 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:02,931 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_02_000000_0] to container: [container_1452789002288_0001_00_000003]
2016-01-14 16:30:02,932 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:02,932 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_02_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:02,932 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:02,932 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, startTime=1452789002932, containerId=container_1452789002288_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:02,933 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:02,933 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:02,933 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000003 given task: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,933 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,934 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,934 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:02,934 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:02,934 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:02,934 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:02,935 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:02,935 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:02,935 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: ECA94FF0DA97475897C9BC9E720E2D3C, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:02,935 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:02,936 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,936 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:02,936 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,938 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C6F906C21EA04AB8AEC5AF372B2F50C8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:02,938 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DDAF1DA7D9CA4B3EBC36522E39432376, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:02,938 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,943 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:02,944 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,943 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:02,944 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,943 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:02,948 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:02,948 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,948 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:02,948 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:02,950 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:02,950 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,949 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:02,951 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,959 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, ordinal: 2
2016-01-14 16:30:02,959 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, mem on start (mb), free: 68, total: 516, max: 672
2016-01-14 16:30:02,959 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:02,960 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:02,960 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:02,960 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:02,960 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:02,960 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:02,961 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:02,961 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:02,961 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:02,961 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DDAF1DA7D9CA4B3EBC36522E39432376, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:02,961 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C6F906C21EA04AB8AEC5AF372B2F50C8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:02,961 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:02,962 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C6F906C21EA04AB8AEC5AF372B2F50C8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:02,962 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,962 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:02,965 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:02,965 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:02,966 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:02,967 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C6F906C21EA04AB8AEC5AF372B2F50C8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:02,967 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:02,967 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:02,967 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:02,968 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,971 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:02,971 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:02,972 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_01_000000_0_10018, runDuration: 160373, ]
2016-01-14 16:30:02,974 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_01_000000_0_10018] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789002974, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:02,974 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,975 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:02,976 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-01-14 16:30:02,976 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [C6F906C21EA04AB8AEC5AF372B2F50C8] #5 in 3ms
2016-01-14 16:30:02,976 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C6F906C21EA04AB8AEC5AF372B2F50C8] #5, status:false, isInterrupted:false
2016-01-14 16:30:02,977 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:02,977 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:02,976 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: C08D4F4BD10C479B9A10DF8909280B38
2016-01-14 16:30:02,977 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:02,978 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:02,978 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:02,978 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:02,978 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:02,979 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:02,979 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:02,983 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@280467e3; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@39407c9b; comparator=cascading.tuple.hadoop.util.TupleComparator@38b4c17a; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:02,983 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:02,984 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:02,984 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,012 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:03,012 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:03,013 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:03,013 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:03,013 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:03,014 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] C08D4F4BD10C479B9A10DF8909280B38, for 1 inputs
2016-01-14 16:30:03,014 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,015 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@52c62ad9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@50273703
2016-01-14 16:30:03,034 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,034 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ECA94FF0DA97475897C9BC9E720E2D3C, mem on close (mb), free: 203, total: 517, max: 672
2016-01-14 16:30:03,034 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C6F906C21EA04AB8AEC5AF372B2F50C8
2016-01-14 16:30:03,034 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:03,035 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:03,035 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,035 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:03,035 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:03,035 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:03,035 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:03,035 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 6291; bufvoid = 66060288
2016-01-14 16:30:03,035 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:03,041 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,044 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:03,045 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789002288_0001_1_02, eventCount=1
2016-01-14 16:30:03,046 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:03,047 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskAttemptId=attempt_1452789002288_0001_1_02_000000_0, startTime=1452789002932, finishTime=1452789003046, timeTaken=114, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0
2016-01-14 16:30:03,047 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:03,047 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03,047 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:03,048 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:03,048 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, taskId=task_1452789002288_0001_1_02_000000, startTime=1452789002932, finishTime=1452789003047, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0
2016-01-14 16:30:03,049 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:03,049 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] : 1
2016-01-14 16:30:03,049 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:03,050 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:03,051 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=ECA94FF0DA97475897C9BC9E720E2D3C, vertexId=vertex_1452789002288_0001_1_02, initRequestedTime=1452789002536, initedTime=1452789002542, startRequestedTime=1452789002596, startedTime=1452789002596, finishTime=1452789003049, timeTaken=453, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0, vertexStats=firstTaskStartTime=1452789002932, firstTasksToStart=[ task_1452789002288_0001_1_02_000000 ], lastTaskFinishTime=1452789003046, lastTasksToFinish=[ task_1452789002288_0001_1_02_000000 ], minTaskDuration=114, maxTaskDuration=114, avgTaskDuration=114.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_02_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:03,051 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:03,051 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=4376, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003014, Process_Duration=20, Process_End_Time=1452789003034, Read_Duration=7, Tuples_Read=174, Tuples_Written=87, Write_Duration=0]]
2016-01-14 16:30:03,052 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:03,052 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000003, stoppedTime=1452789003052, exitStatus=0
2016-01-14 16:30:03,052 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:03,052 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:03,052 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] attempt: attempt_1452789002288_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:03,052 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DDAF1DA7D9CA4B3EBC36522E39432376
2016-01-14 16:30:03,053 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,053 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:03,054 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789002288_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:03,054 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:03,054 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,054 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,055 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:03,055 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_STARTED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskId=task_1452789002288_0001_1_03_000000, scheduledTime=1452789003054, launchTime=1452789003054
2016-01-14 16:30:03,055 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:03,055 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:03,056 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:03,056 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000003, Exiting
2016-01-14 16:30:03,056 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,056 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000003
2016-01-14 16:30:03,056 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:03,057 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,057 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:03,057 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: DDAF1DA7D9CA4B3EBC36522E39432376, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:03,057 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000003 completed successfully
2016-01-14 16:30:03,057 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:03,058 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:03,058 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:03,059 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789002288_0001_00_000004
2016-01-14 16:30:03,059 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:03,061 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789002288_0001_00_000004
2016-01-14 16:30:03,062 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:03,066 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789002288_0001_00_000004, launchTime=1452789003066
2016-01-14 16:30:03,066 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:03,066 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,067 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:03,067 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789002288_0001_1_03_000000_0] to container: [container_1452789002288_0001_00_000004]
2016-01-14 16:30:03,067 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:03,068 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789002288_0001_00_000004 given task: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,068 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,068 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789002288_0001_1_03_000000_0] started. Is using containerId: [container_1452789002288_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:03,069 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,069 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,069 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:03,069 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, startTime=1452789003068, containerId=container_1452789002288_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789002288_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:03,069 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:03,070 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:03,070 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:03,070 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:03,070 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:03,070 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB] pearson/(1/1) ...mposite/pearson/composite, VertexName: DDAF1DA7D9CA4B3EBC36522E39432376, VertexParallelism: 1, TaskAttemptID:attempt_1452789002288_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:03,070 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:03,071 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:03,071 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:03,072 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789002288_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:03,072 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,072 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:03,073 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,075 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=ECA94FF0DA97475897C9BC9E720E2D3C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:03,076 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=668F7BA666CF49C2884A386FFBEAC8AF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:03,076 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:03,075 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,077 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:03,077 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,077 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:03,078 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,080 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:03,084 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,086 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:03,086 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:03,087 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,088 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,100 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:03,100 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,102 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, ordinal: 3
2016-01-14 16:30:03,102 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, mem on start (mb), free: 194, total: 517, max: 672
2016-01-14 16:30:03,102 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,102 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:03,103 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:03,103 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:03,104 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:03,104 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:03,104 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:03,105 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:03,105 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:03,106 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:03,106 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 668F7BA666CF49C2884A386FFBEAC8AF, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:03,106 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, ECA94FF0DA97475897C9BC9E720E2D3C, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:03,107 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:03,107 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: ECA94FF0DA97475897C9BC9E720E2D3C being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:03,107 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,108 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:03,108 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:03,108 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,109 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:03,109 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: ECA94FF0DA97475897C9BC9E720E2D3C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:03,109 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:03,109 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:03,109 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:03,110 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,111 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:03,111 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,112 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789002288_0001_1_02_000000_0_10021, runDuration: 98756, ]
2016-01-14 16:30:03,114 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-01-14 16:30:03,114 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789002288_0001_1_02_000000_0_10021] to DISK_DIRECT, CompressedSize=4344, DecompressedSize=4340,EndTime=1452789003114, TimeTaken=1, Rate=4.14 MB/s
2016-01-14 16:30:03,114 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,114 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:03,115 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [ECA94FF0DA97475897C9BC9E720E2D3C] #6 in 2ms
2016-01-14 16:30:03,115 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: 309E7276ABA34241BC798F15E92A990D
2016-01-14 16:30:03,115 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [ECA94FF0DA97475897C9BC9E720E2D3C] #6, status:false, isInterrupted:false
2016-01-14 16:30:03,116 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:03,116 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 4344 bytes from disk
2016-01-14 16:30:03,116 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,116 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:03,117 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: 668F7BA666CF49C2884A386FFBEAC8AF
2016-01-14 16:30:03,117 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-01-14 16:30:03,117 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,117 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-01-14 16:30:03,119 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:03,119 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] 309E7276ABA34241BC798F15E92A990D, for 1 inputs
2016-01-14 16:30:03,119 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,120 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@56f10386; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@691e7f01
2016-01-14 16:30:03,138 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:03,138 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:03,139 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,139 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789002288_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:03,140 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,140 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DDAF1DA7D9CA4B3EBC36522E39432376, mem on close (mb), free: 191, total: 517, max: 672
2016-01-14 16:30:03,141 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: ECA94FF0DA97475897C9BC9E720E2D3C
2016-01-14 16:30:03,141 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:03,142 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:03,142 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:03,142 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:03,142 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,144 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:03,145 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:03,146 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:03,146 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,146 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskAttemptId=attempt_1452789002288_0001_1_03_000000_0, startTime=1452789003068, finishTime=1452789003145, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:03,146 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:03,146 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789002288_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:03,146 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03,146 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,147 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:03,147 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:TASK_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, taskId=task_1452789002288_0001_1_03_000000, startTime=1452789003068, finishTime=1452789003146, timeTaken=78, status=SUCCEEDED, successfulAttemptID=attempt_1452789002288_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:03,148 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789002288_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:03,148 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:03,148 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] : 1
2016-01-14 16:30:03,148 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,149 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:03,150 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:VERTEX_FINISHED]: vertexName=DDAF1DA7D9CA4B3EBC36522E39432376, vertexId=vertex_1452789002288_0001_1_03, initRequestedTime=1452789002548, initedTime=1452789002585, startRequestedTime=1452789002603, startedTime=1452789002603, finishTime=1452789003148, timeTaken=545, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=4376, FILE_BYTES_WRITTEN=1369, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542113792, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003120, Process_Duration=20, Process_End_Time=1452789003140, Read_Duration=11, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789003068, firstTasksToStart=[ task_1452789002288_0001_1_03_000000 ], lastTaskFinishTime=1452789003145, lastTasksToFinish=[ task_1452789002288_0001_1_03_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789002288_0001_1_03_000000 ], longestDurationTasks=[ task_1452789002288_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:03,150 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:03,150 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:03,150 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:03,150 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789002288_0001_00_000004, stoppedTime=1452789003150, exitStatus=0
2016-01-14 16:30:03,151 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:03,151 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:03,151 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789002288_0001_1
2016-01-14 16:30:03,151 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789002288_0001_1
2016-01-14 16:30:03,151 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_00 [A70FD1D546294A2AB0C612D3B44665F7]
2016-01-14 16:30:03,151 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_01 [C6F906C21EA04AB8AEC5AF372B2F50C8]
2016-01-14 16:30:03,152 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789002288_0001_1_02 [ECA94FF0DA97475897C9BC9E720E2D3C]
2016-01-14 16:30:03,152 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 668F7BA666CF49C2884A386FFBEAC8AF for vertex: vertex_1452789002288_0001_1_03 [DDAF1DA7D9CA4B3EBC36522E39432376]
2016-01-14 16:30:03,157 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789002288_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:03,157 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789002288_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:03,157 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789002288_0001_00_000004, Exiting
2016-01-14 16:30:03,157 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,157 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789002288_0001_00_000004
2016-01-14 16:30:03,158 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789002288_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:03,159 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789002288_0001_00_000004 completed successfully
2016-01-14 16:30:03,166 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789002288_0001_1, queueSize=0
2016-01-14 16:30:03,167 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:03,167 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789002288_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789002288_0001_1, startTime=1452789002524, finishTime=1452789003161, timeTaken=637, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=14386, FILE_BYTES_WRITTEN=14937, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2167406592, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=20, MERGE_PHASE_TIME=28, FIRST_EVENT_RECEIVED=14, LAST_EVENT_RECEIVED=14, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156011669, Process_Duration=74, Process_End_Time=5811156011743, Read_Duration=21, Tuples_Read=303, Tuples_Written=317, Write_Duration=6, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:03,167 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789002288_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:03,168 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789002288_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:03,168 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452789002288_0001_00_000004
2016-01-14 16:30:03,169 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789002288_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:03 Completed Dag: dag_1452789002288_0001_1
2016-01-14 16:30:03,170 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789002288_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:03,170 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789002288_0001_1_post
2016-01-14 16:30:03,170 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,170 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:03,177 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1452789002288_0001
2016-01-14 16:30:03,180 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:03,180 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:03,180 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:03,180 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:03,180 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:03,181 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:03,181 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:03,181 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:03,181 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:03,182 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:03,182 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:03,182 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:03,184 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:03,188 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/85126C688C114D22A926C44DE9FFBDF4/D18176B25E7C4784944EBC1586418CBB/.tez/application_1452789002288_0001
2016-01-14 16:30:03,189 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [pearson]  completed in: 00:00.930
2016-01-14 16:30:03,270 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-01-14 16:30:03,303 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:03,304 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:03,386 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.051
2016-01-14 16:30:03,416 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.080
2016-01-14 16:30:03,417 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:03,417 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:03,417 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:03,440 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:03,475 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 16:30:03,623 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:30:03,623 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,624 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 16:30:03,624 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 16:30:03,624 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 16:30:03,625 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 16:30:03,626 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,626 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:03,642 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:30:03,647 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:03,647 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:03,648 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:03,658 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001 doesn't exist and is created
2016-01-14 16:30:03,677 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:03,677 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:03,678 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001
2016-01-14 16:30:03,683 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789003647_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:03,696 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:03,697 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:03,699 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:03,699 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:03,700 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir
2016-01-14 16:30:03,700 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/history.txt.appattempt_1452789003647_0001_000000, maxErrors=10
2016-01-14 16:30:03,700 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:03,700 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789003647_0001_000000, appSubmitTime=1452789003682, launchTime=1452789003682
2016-01-14 16:30:03,707 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:03,708 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:03,713 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:03,713 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:03,714 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:52802
2016-01-14 16:30:03,717 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:03,719 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789003647_0001_000000, startTime=1452789003719
2016-01-14 16:30:03,719 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:03,719 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:03,778 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:03,778 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:03,778 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001, dagName=[88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,797 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,799 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789003647_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/dag_1452789003647_0001_1.dot
2016-01-14 16:30:03,801 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001/localmode-log-dir/dag_1452789003647_0001_1-tez-dag.pb.txt
2016-01-14 16:30:03,838 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789003647_0001_1
2016-01-14 16:30:03,839 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,839 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03 Running Dag: dag_1452789003647_0001_1
2016-01-14 16:30:03,856 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789003647_0001_1, submitTime=1452789003798
2016-01-14 16:30:03,856 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:03,857 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:03,857 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:03,858 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789003647_0001_1, initTime=1452789003856
2016-01-14 16:30:03,858 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:03,858 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:03,859 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_STARTED]: dagID=dag_1452789003647_0001_1, startTime=1452789003859
2016-01-14 16:30:03,860 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:03,860 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:03,860 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: ABC4B209A76B4457A915432747B60C7F : {79D4628DF96A475FA05895612AA09065={InputName=79D4628DF96A475FA05895612AA09065}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:03,860 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 79D4628DF96A475FA05895612AA09065, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:03,860 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,861 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]. Starting root input initializers: 1
2016-01-14 16:30:03,862 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001, dagName=[88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long
2016-01-14 16:30:03,862 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:03,862 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 79D4628DF96A475FA05895612AA09065 on vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,862 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,862 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:03,865 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:03,865 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 79D4628DF96A475FA05895612AA09065 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:03,868 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,868 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:03,868 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,869 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:03,869 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,869 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, vertexId=vertex_1452789003647_0001_1_01, initRequestedTime=1452789003862, initedTime=1452789003869, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,870 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,870 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,868 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:03,875 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:03,875 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,875 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 79D4628DF96A475FA05895612AA09065 on vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,876 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,876 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:03,876 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,878 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, vertexId=vertex_1452789003647_0001_1_02, initRequestedTime=1452789003870, initedTime=1452789003876, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,878 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,878 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:03,879 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {79D4628DF96A475FA05895612AA09065=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:03,879 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] parallelism set to 1
2016-01-14 16:30:03,879 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:03,880 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_00, eventCount=1
2016-01-14 16:30:03,880 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ABC4B209A76B4457A915432747B60C7F, vertexId=vertex_1452789003647_0001_1_00, initRequestedTime=1452789003860, initedTime=1452789003880, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:03,881 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,881 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] with distanceFromRoot: 0
2016-01-14 16:30:03,881 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:03,881 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,886 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:03,886 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,887 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:03,887 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,888 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452789003647_0001, with dag id: dag_1452789003647_0001_1
2016-01-14 16:30:03,891 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=0BA133967AE44E27A765D92F7FE5973B, vertexName=FDBD1DD030994713ADFA688DAB68291D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:03,891 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=0BA133967AE44E27A765D92F7FE5973B, vertexId=vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:03,893 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, vertexId=vertex_1452789003647_0001_1_03, initRequestedTime=1452789003881, initedTime=1452789003893, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:03,893 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:03,893 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:03,893 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:03,894 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,894 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_00, startRequestedTime=1452789003893, startedTime=1452789003893
2016-01-14 16:30:03,894 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,895 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,897 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:03,897 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskId=task_1452789003647_0001_1_00_000000, scheduledTime=1452789003897, launchTime=1452789003897
2016-01-14 16:30:03,897 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:03,898 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_00 for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,900 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] with distanceFromRoot: 1
2016-01-14 16:30:03,900 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:03,901 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: ABC4B209A76B4457A915432747B60C7F in vertex: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,901 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: B850D941A7724568A46A0CC2FE9491F8 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,902 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,902 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_01, startRequestedTime=1452789003901, startedTime=1452789003901
2016-01-14 16:30:03,902 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,903 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: ABC4B209A76B4457A915432747B60C7F, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:03,907 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:03,908 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_01 for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,909 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] with distanceFromRoot: 2
2016-01-14 16:30:03,910 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:03,910 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B850D941A7724568A46A0CC2FE9491F8 in vertex: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:03,911 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D0E9805FF68B45CAB52AC3539931AA0A with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,911 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,911 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_02, startRequestedTime=1452789003910, startedTime=1452789003910
2016-01-14 16:30:03,911 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,911 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:03,912 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789003647_0001_1_02 for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] numStartedSources: 1 numSources: 1
2016-01-14 16:30:03,912 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] with distanceFromRoot: 3
2016-01-14 16:30:03,913 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:03,913 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:03,914 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000001
2016-01-14 16:30:03,915 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:03,915 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D0E9805FF68B45CAB52AC3539931AA0A in vertex: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:03,915 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: FDBD1DD030994713ADFA688DAB68291D with 1 source tasks and 1 pending tasks
2016-01-14 16:30:03,915 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:03,916 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789003647_0001_1_03, startRequestedTime=1452789003915, startedTime=1452789003915
2016-01-14 16:30:03,916 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:03,918 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789003647_0001_00_000001
2016-01-14 16:30:03,918 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:03,919 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:03,919 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:03,920 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000001, launchTime=1452789003919
2016-01-14 16:30:03,921 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000001
2016-01-14 16:30:03,922 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_00_000000_0] to container: [container_1452789003647_0001_00_000001]
2016-01-14 16:30:03,922 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:03,923 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000001 given task: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_00_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,923 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:03,924 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:03,924 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:03,924 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:03,924 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, startTime=1452789003923, containerId=container_1452789003647_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:03,925 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:03,924 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:03,925 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:03,926 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:03,926 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: ABC4B209A76B4457A915432747B60C7F, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:03,927 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:03,925 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:03,928 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,928 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:03,928 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,929 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,931 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=79D4628DF96A475FA05895612AA09065, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:03,933 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:03,933 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,934 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,934 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:03,931 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:03,936 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:03,936 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,935 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:03,938 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:03,938 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,938 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,939 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:03,939 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,940 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:03,949 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ABC4B209A76B4457A915432747B60C7F, ordinal: 0
2016-01-14 16:30:03,949 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABC4B209A76B4457A915432747B60C7F, mem on start (mb), free: 54, total: 517, max: 672
2016-01-14 16:30:03,950 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:03,950 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:03,950 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:03,950 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:03,950 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:03,951 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:03,951 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:03,951 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:03,952 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:03,952 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:03,952 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 79D4628DF96A475FA05895612AA09065, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:03,953 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B850D941A7724568A46A0CC2FE9491F8, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:03,953 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:03,953 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 79D4628DF96A475FA05895612AA09065 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:03,953 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,954 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:03,953 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:03,954 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:03,954 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,955 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:03,955 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,956 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:03,957 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@378095fb
2016-01-14 16:30:03,959 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@378095fb
2016-01-14 16:30:03,959 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:03,960 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:03,961 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,962 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: 79D4628DF96A475FA05895612AA09065
2016-01-14 16:30:03,962 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:03,962 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:03,963 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6d47d006; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@5aa71f7; comparator=cascading.tuple.hadoop.util.TupleComparator@79201208; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:03,964 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:03,964 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:03,964 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:03,990 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:03,990 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:03,991 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:03,991 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:03,991 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:03,993 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:03,994 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:03,994 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ABC4B209A76B4457A915432747B60C7F, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:03,997 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:03,998 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABC4B209A76B4457A915432747B60C7F, mem on close (mb), free: 161, total: 516, max: 672
2016-01-14 16:30:03,998 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:03,998 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:03,998 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:03,998 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:04,001 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04,007 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,008 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_00, eventCount=1
2016-01-14 16:30:04,008 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,008 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskAttemptId=attempt_1452789003647_0001_1_00_000000_0, startTime=1452789003923, finishTime=1452789004008, timeTaken=85, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:04,008 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,009 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,009 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04,010 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, taskId=task_1452789003647_0001_1_00_000000, startTime=1452789003923, finishTime=1452789004009, timeTaken=86, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:04,010 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,010 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,010 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] : 1
2016-01-14 16:30:04,010 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:04,011 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,012 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=ABC4B209A76B4457A915432747B60C7F, vertexId=vertex_1452789003647_0001_1_00, initRequestedTime=1452789003860, initedTime=1452789003880, startRequestedTime=1452789003893, startedTime=1452789003893, finishTime=1452789004010, timeTaken=117, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789003923, firstTasksToStart=[ task_1452789003647_0001_1_00_000000 ], lastTaskFinishTime=1452789004008, lastTasksToFinish=[ task_1452789003647_0001_1_00_000000 ], minTaskDuration=85, maxTaskDuration=85, avgTaskDuration=85.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_00_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,012 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,012 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,012 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000001, stoppedTime=1452789004012, exitStatus=0
2016-01-14 16:30:04,013 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,013 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1641, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=12, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789003994, Process_Duration=3, Process_End_Time=1452789003997, Read_Duration=0, Tuples_Read=7, Tuples_Written=35, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:04,013 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] attempt: attempt_1452789003647_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,012 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,013 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,013 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:04,014 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,014 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,014 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,015 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,015 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,015 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000001, Exiting
2016-01-14 16:30:04,015 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,015 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000001
2016-01-14 16:30:04,015 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,016 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,017 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,017 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskId=task_1452789003647_0001_1_01_000000, scheduledTime=1452789004016, launchTime=1452789004016
2016-01-14 16:30:04,017 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,018 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000001 completed successfully
2016-01-14 16:30:04,018 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,018 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,018 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,018 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,019 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B850D941A7724568A46A0CC2FE9491F8, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,019 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,019 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,020 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000002
2016-01-14 16:30:04,020 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,022 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789003647_0001_00_000002
2016-01-14 16:30:04,023 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,025 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,025 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,025 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000002, launchTime=1452789004025
2016-01-14 16:30:04,026 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,027 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_01_000000_0] to container: [container_1452789003647_0001_00_000002]
2016-01-14 16:30:04,027 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,027 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000002 given task: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,028 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_01_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,028 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,028 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,028 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, startTime=1452789004028, containerId=container_1452789003647_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:04,029 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,029 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,029 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,030 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,030 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,030 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,030 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,030 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,031 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,031 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: B850D941A7724568A46A0CC2FE9491F8, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,031 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,032 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,032 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,032 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,033 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=ABC4B209A76B4457A915432747B60C7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,033 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,034 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,034 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,036 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:04,036 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:04,037 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,037 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,038 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,038 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,038 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,039 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,039 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,040 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:04,040 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,041 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,048 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, ordinal: 1
2016-01-14 16:30:04,048 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, mem on start (mb), free: 153, total: 516, max: 672
2016-01-14 16:30:04,048 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,048 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,048 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,048 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,048 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,049 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,049 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:04,049 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,049 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:04,050 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, ABC4B209A76B4457A915432747B60C7F, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:04,050 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D0E9805FF68B45CAB52AC3539931AA0A, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:04,050 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,051 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: ABC4B209A76B4457A915432747B60C7F being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,051 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,051 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,053 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,054 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,054 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,055 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: ABC4B209A76B4457A915432747B60C7F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,055 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:04,056 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,056 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,060 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,060 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,061 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,063 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 16:30:04,064 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 0016BEC6C7F04E1DA64780FB4A4C80A3
2016-01-14 16:30:04,064 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,064 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,064 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_00_000000_0_10027, runDuration: 68811, ]
2016-01-14 16:30:04,065 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@70e793d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@47399769; comparator=cascading.tuple.hadoop.util.TupleComparator@16039307; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:04,065 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:04,066 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:04,067 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,067 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_00_000000_0_10027] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789004067, TimeTaken=2, Rate=0.52 MB/s
2016-01-14 16:30:04,068 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,068 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:04,068 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [ABC4B209A76B4457A915432747B60C7F] #7 in 3ms
2016-01-14 16:30:04,069 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [ABC4B209A76B4457A915432747B60C7F] #7, status:false, isInterrupted:false
2016-01-14 16:30:04,076 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:04,078 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:04,078 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:04,078 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:04,078 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:04,076 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,079 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:04,079 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0016BEC6C7F04E1DA64780FB4A4C80A3, for 1 inputs
2016-01-14 16:30:04,079 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,079 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:04,079 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:04,079 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,080 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:04,080 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,081 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:04,081 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,082 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3376b54c; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@3678aaf9
2016-01-14 16:30:04,111 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,111 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B850D941A7724568A46A0CC2FE9491F8, mem on close (mb), free: 87, total: 516, max: 672
2016-01-14 16:30:04,111 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: ABC4B209A76B4457A915432747B60C7F
2016-01-14 16:30:04,111 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,112 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,112 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,112 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,112 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04,112 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:04,113 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:04,113 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:04,113 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:04,127 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,133 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,134 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_01, eventCount=1
2016-01-14 16:30:04,135 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,135 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-01-14 16:30:04,136 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskAttemptId=attempt_1452789003647_0001_1_01_000000_0, startTime=1452789004028, finishTime=1452789004134, timeTaken=106, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 16:30:04,136 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:04,137 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,138 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,138 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04,138 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,138 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,138 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,139 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, taskId=task_1452789003647_0001_1_01_000000, startTime=1452789004028, finishTime=1452789004138, timeTaken=110, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-01-14 16:30:04,139 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,139 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] : 1
2016-01-14 16:30:04,139 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,141 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,141 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=B850D941A7724568A46A0CC2FE9491F8, vertexId=vertex_1452789003647_0001_1_01, initRequestedTime=1452789003862, initedTime=1452789003869, startRequestedTime=1452789003901, startedTime=1452789003901, finishTime=1452789004140, timeTaken=239, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541065216, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004081, Process_Duration=30, Process_End_Time=1452789004111, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1452789004028, firstTasksToStart=[ task_1452789003647_0001_1_01_000000 ], lastTaskFinishTime=1452789004134, lastTasksToFinish=[ task_1452789003647_0001_1_01_000000 ], minTaskDuration=106, maxTaskDuration=106, avgTaskDuration=106.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_01_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,142 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,142 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,142 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] attempt: attempt_1452789003647_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,142 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,142 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,143 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000002, stoppedTime=1452789004143, exitStatus=0
2016-01-14 16:30:04,143 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,143 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,143 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,144 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskId=task_1452789003647_0001_1_02_000000, scheduledTime=1452789004144, launchTime=1452789004144
2016-01-14 16:30:04,144 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,144 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,144 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,145 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,145 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,146 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D0E9805FF68B45CAB52AC3539931AA0A, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,146 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,147 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,147 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000003
2016-01-14 16:30:04,148 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,150 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789003647_0001_00_000003
2016-01-14 16:30:04,150 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - TezTask metrics system already initialized!
2016-01-14 16:30:04,151 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,151 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000002: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,151 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,151 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,153 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000002, Exiting
2016-01-14 16:30:04,153 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,153 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000002
2016-01-14 16:30:04,154 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000003, launchTime=1452789004151
2016-01-14 16:30:04,154 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,154 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000002 completed successfully
2016-01-14 16:30:04,154 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,156 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,157 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_02_000000_0] to container: [container_1452789003647_0001_00_000003]
2016-01-14 16:30:04,157 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,158 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_02_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,159 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, startTime=1452789004158, containerId=container_1452789003647_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:04,159 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,159 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,159 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04,160 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000003 given task: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,161 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,162 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,162 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,162 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,162 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,162 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,163 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,163 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,163 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D0E9805FF68B45CAB52AC3539931AA0A, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:04,163 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,166 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,166 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,166 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,168 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B850D941A7724568A46A0CC2FE9491F8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,169 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,169 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,171 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,171 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=FDBD1DD030994713ADFA688DAB68291D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:04,172 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,172 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,176 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:04,176 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,178 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:04,178 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,179 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,181 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,182 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,182 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,184 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,188 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, ordinal: 2
2016-01-14 16:30:04,188 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, mem on start (mb), free: 81, total: 516, max: 672
2016-01-14 16:30:04,188 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,188 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,188 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,188 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,188 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,189 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:04,190 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,191 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,191 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:04,192 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, FDBD1DD030994713ADFA688DAB68291D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:04,193 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B850D941A7724568A46A0CC2FE9491F8, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:04,193 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,193 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B850D941A7724568A46A0CC2FE9491F8 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,193 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,195 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,195 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,195 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,196 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,198 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B850D941A7724568A46A0CC2FE9491F8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,198 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:04,198 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,198 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,200 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,201 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,201 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,203 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_01_000000_0_10030, runDuration: 91682, ]
2016-01-14 16:30:04,204 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:04,205 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 3F18F079E0944129A47CF89B52319255
2016-01-14 16:30:04,207 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,207 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,207 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_01_000000_0_10030] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789004207, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:04,209 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,208 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@2eec5c0d; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@1f8f718c; comparator=cascading.tuple.hadoop.util.TupleComparator@267ae84; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:04,210 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:04,210 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:04,210 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,209 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:04,221 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [B850D941A7724568A46A0CC2FE9491F8] #8 in 16ms
2016-01-14 16:30:04,221 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B850D941A7724568A46A0CC2FE9491F8] #8, status:false, isInterrupted:false
2016-01-14 16:30:04,229 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:04,230 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,230 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:04,230 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,231 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,231 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:04,231 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:04,232 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:04,232 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:04,231 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:04,232 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,233 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,234 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 3F18F079E0944129A47CF89B52319255, for 1 inputs
2016-01-14 16:30:04,235 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:04,235 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1984599; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@340140a0
2016-01-14 16:30:04,255 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,256 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0E9805FF68B45CAB52AC3539931AA0A, mem on close (mb), free: 200, total: 516, max: 672
2016-01-14 16:30:04,256 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B850D941A7724568A46A0CC2FE9491F8
2016-01-14 16:30:04,257 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,257 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,258 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,258 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,258 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04,259 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:04,259 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:04,259 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 16:30:04,260 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:04,265 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,269 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,270 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789003647_0001_1_02, eventCount=1
2016-01-14 16:30:04,271 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,272 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskAttemptId=attempt_1452789003647_0001_1_02_000000_0, startTime=1452789004158, finishTime=1452789004270, timeTaken=112, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-01-14 16:30:04,272 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,272 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,272 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04,273 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,273 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, taskId=task_1452789003647_0001_1_02_000000, startTime=1452789004158, finishTime=1452789004273, timeTaken=115, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-01-14 16:30:04,274 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,274 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] : 1
2016-01-14 16:30:04,274 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,275 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,276 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=D0E9805FF68B45CAB52AC3539931AA0A, vertexId=vertex_1452789003647_0001_1_02, initRequestedTime=1452789003870, initedTime=1452789003876, startRequestedTime=1452789003910, startedTime=1452789003910, finishTime=1452789004274, timeTaken=364, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4, vertexStats=firstTaskStartTime=1452789004158, firstTasksToStart=[ task_1452789003647_0001_1_02_000000 ], lastTaskFinishTime=1452789004270, lastTasksToFinish=[ task_1452789003647_0001_1_02_000000 ], minTaskDuration=112, maxTaskDuration=112, avgTaskDuration=112.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_02_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,276 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,276 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,276 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000003, stoppedTime=1452789004276, exitStatus=0
2016-01-14 16:30:04,276 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,276 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,277 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] attempt: attempt_1452789003647_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:04,277 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FDBD1DD030994713ADFA688DAB68291D
2016-01-14 16:30:04,277 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,277 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,277 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789003647_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:04,278 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_STARTED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskId=task_1452789003647_0001_1_03_000000, scheduledTime=1452789004278, launchTime=1452789004278
2016-01-14 16:30:04,278 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:04,278 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=25, MERGE_PHASE_TIME=36, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004235, Process_Duration=20, Process_End_Time=1452789004255, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=4]]
2016-01-14 16:30:04,278 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:04,279 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,279 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:04,279 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FDBD1DD030994713ADFA688DAB68291D, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:04,279 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:04,280 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,280 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,279 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:04,280 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,280 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:04,280 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,281 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000003, Exiting
2016-01-14 16:30:04,281 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,281 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000003
2016-01-14 16:30:04,281 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,282 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000003 completed successfully
2016-01-14 16:30:04,282 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789003647_0001_00_000004
2016-01-14 16:30:04,282 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:04,283 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04,284 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789003647_0001_00_000004
2016-01-14 16:30:04,285 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,290 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789003647_0001_00_000004, launchTime=1452789004290
2016-01-14 16:30:04,290 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:04,291 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,292 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:04,293 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789003647_0001_1_03_000000_0] to container: [container_1452789003647_0001_00_000004]
2016-01-14 16:30:04,293 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:04,293 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789003647_0001_00_000004 given task: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,293 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,294 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789003647_0001_1_03_000000_0] started. Is using containerId: [container_1452789003647_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:04,294 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,294 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,294 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, startTime=1452789004294, containerId=container_1452789003647_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789003647_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:04,294 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:04,294 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:04,295 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:04,295 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:04,295 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:04,295 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:04,295 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:04,296 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: FDBD1DD030994713ADFA688DAB68291D, VertexParallelism: 1, TaskAttemptID:attempt_1452789003647_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:04,296 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:04,297 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:04,298 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,298 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:04,299 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,299 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789003647_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:04,299 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D0E9805FF68B45CAB52AC3539931AA0A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:04,300 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:04,300 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,301 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,301 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=0BA133967AE44E27A765D92F7FE5973B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:04,302 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:04,302 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,303 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:04,303 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,304 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:04,309 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:04,309 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:04,309 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,317 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,319 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:04,319 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,323 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, ordinal: 3
2016-01-14 16:30:04,323 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, mem on start (mb), free: 193, total: 516, max: 672
2016-01-14 16:30:04,323 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:04,323 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:04,323 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:04,324 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:04,324 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:04,324 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:04,324 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:04,324 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:04,324 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:04,325 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:04,325 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 0BA133967AE44E27A765D92F7FE5973B, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:04,325 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D0E9805FF68B45CAB52AC3539931AA0A, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:04,325 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:04,325 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D0E9805FF68B45CAB52AC3539931AA0A being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:04,325 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,326 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:04,326 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:04,326 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:04,327 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:04,327 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D0E9805FF68B45CAB52AC3539931AA0A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:04,327 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:04,328 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:04,328 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:04,328 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,329 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:04,330 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,332 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789003647_0001_1_02_000000_0_10033, runDuration: 95380, ]
2016-01-14 16:30:04,334 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 16:30:04,335 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 264191978B304D7DA6FEC6B650FF46F5
2016-01-14 16:30:04,335 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: 0BA133967AE44E27A765D92F7FE5973B
2016-01-14 16:30:04,335 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789003647_0001_1_02_000000_0_10033] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452789004335, TimeTaken=2, Rate=2.44 MB/s
2016-01-14 16:30:04,336 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,336 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:04,336 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-01-14 16:30:04,336 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D0E9805FF68B45CAB52AC3539931AA0A] #9 in 3ms
2016-01-14 16:30:04,336 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D0E9805FF68B45CAB52AC3539931AA0A] #9, status:false, isInterrupted:false
2016-01-14 16:30:04,337 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 264191978B304D7DA6FEC6B650FF46F5, for 1 inputs
2016-01-14 16:30:04,337 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:04,338 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 16:30:04,338 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,338 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:04,338 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 16:30:04,338 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,339 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:04,339 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, all 1 inputs ready in: 00:00:00.002
2016-01-14 16:30:04,340 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@4d7c5890; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@1e9a7dae
2016-01-14 16:30:04,348 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:04,348 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:04,349 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,349 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789003647_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:04,350 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,350 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FDBD1DD030994713ADFA688DAB68291D, mem on close (mb), free: 190, total: 516, max: 672
2016-01-14 16:30:04,350 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D0E9805FF68B45CAB52AC3539931AA0A
2016-01-14 16:30:04,351 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:04,351 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:04,351 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:04,351 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:04,352 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,352 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:04,353 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:04,354 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskAttemptId=attempt_1452789003647_0001_1_03_000000_0, startTime=1452789004294, finishTime=1452789004353, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:04,353 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:04,355 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:04,355 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,355 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,354 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789003647_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:04,356 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:04,356 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04,357 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:04,357 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:TASK_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, taskId=task_1452789003647_0001_1_03_000000, startTime=1452789004294, finishTime=1452789004356, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1452789003647_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:04,357 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789003647_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:04,357 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] : 1
2016-01-14 16:30:04,357 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,358 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:04,359 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:VERTEX_FINISHED]: vertexName=FDBD1DD030994713ADFA688DAB68291D, vertexId=vertex_1452789003647_0001_1_03, initRequestedTime=1452789003881, initedTime=1452789003893, startRequestedTime=1452789003915, startedTime=1452789003915, finishTime=1452789004357, timeTaken=442, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=541589504, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789004339, Process_Duration=11, Process_End_Time=1452789004350, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789004294, firstTasksToStart=[ task_1452789003647_0001_1_03_000000 ], lastTaskFinishTime=1452789004353, lastTasksToFinish=[ task_1452789003647_0001_1_03_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789003647_0001_1_03_000000 ], longestDurationTasks=[ task_1452789003647_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:04,359 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:04,359 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:04,359 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789003647_0001_00_000004, stoppedTime=1452789004359, exitStatus=0
2016-01-14 16:30:04,359 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:04,359 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:04,359 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:04,360 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789003647_0001_1
2016-01-14 16:30:04,360 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789003647_0001_1
2016-01-14 16:30:04,360 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_02 [D0E9805FF68B45CAB52AC3539931AA0A]
2016-01-14 16:30:04,360 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 0BA133967AE44E27A765D92F7FE5973B for vertex: vertex_1452789003647_0001_1_03 [FDBD1DD030994713ADFA688DAB68291D]
2016-01-14 16:30:04,366 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789003647_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:04,366 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789003647_0001_00_000004: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:04,366 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789003647_0001_00_000004, Exiting
2016-01-14 16:30:04,367 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,367 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789003647_0001_00_000004
2016-01-14 16:30:04,368 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789003647_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:04,368 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789003647_0001_00_000004 completed successfully
2016-01-14 16:30:04,370 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_01 [B850D941A7724568A46A0CC2FE9491F8]
2016-01-14 16:30:04,370 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789003647_0001_1_00 [ABC4B209A76B4457A915432747B60C7F]
2016-01-14 16:30:04,374 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789003647_0001_1, queueSize=0
2016-01-14 16:30:04,374 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:04,374 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789003647_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789003647_0001_1, startTime=1452789003859, finishTime=1452789004370, timeTaken=511, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15771, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=22, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2165309440, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=47, MERGE_PHASE_TIME=72, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156016649, Process_Duration=64, Process_End_Time=5811156016713, Read_Duration=10, Tuples_Read=303, Tuples_Written=317, Write_Duration=12, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:04,374 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789003647_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:04,375 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789003647_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:04,375 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1452789003647_0001_00_000004
2016-01-14 16:30:04,375 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789003647_0001_00_000004 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-01-14 16:30:04 Completed Dag: dag_1452789003647_0001_1
2016-01-14 16:30:04,375 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789003647_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:04,375 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789003647_0001_1_post
2016-01-14 16:30:04,375 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,376 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:04,383 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1452789003647_0001
2016-01-14 16:30:04,386 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:04,386 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:04,386 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:04,386 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:04,386 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:04,387 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:04,388 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:04,388 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:04,388 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:04,389 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:04,389 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:04,390 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:04,390 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:04,397 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/88FFD1D2162C4E0D82784B0774D5F016/E105391B96D745348DD33FA386E229AD/.tez/application_1452789003647_0001
2016-01-14 16:30:04,398 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:00.774
2016-01-14 16:30:04,479 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-01-14 16:30:04,510 INFO  util.Util (Util.java:findMainClass(1327)) - using default application jar, may cause class not found exceptions on the cluster
2016-01-14 16:30:04,511 INFO  planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(149)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.0.4/561447ee876073bb61fddc52e2ad157f45c8edd/cascading-hadoop2-tez-3.0.4.jar
2016-01-14 16:30:04,581 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.042
2016-01-14 16:30:04,603 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.064
2016-01-14 16:30:04,604 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:04,604 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-01-14 16:30:04,604 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-01-14 16:30:04,624 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt
2016-01-14 16:30:04,647 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-01-14 16:30:04,784 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting
2016-01-14 16:30:04,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:04,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 16:30:04,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  parallel execution of steps is enabled: false
2016-01-14 16:30:04,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  executing total steps: 1
2016-01-14 16:30:04,785 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  allocating management threads: 1
2016-01-14 16:30:04,786 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,786 WARN  flow.Flow (BaseFlow.java:logWarn(1445)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-01-14 16:30:04,799 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] tez session mode enabled: true
2016-01-14 16:30:04,804 INFO  client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:04,804 INFO  client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-01-14 16:30:04,804 INFO  client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since  'tez.ignore.lib.uris' is set to true
2016-01-14 16:30:04,814 INFO  client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001 doesn't exist and is created
2016-01-14 16:30:04,831 INFO  client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-01-14 16:30:04,831 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-01-14 16:30:04,831 INFO  client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001
2016-01-14 16:30:04,835 INFO  app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1452789004804_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-01-14 16:30:04,846 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-01-14 16:30:04,847 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-01-14 16:30:04,849 INFO  node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-01-14 16:30:04,849 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-01-14 16:30:04,849 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir
2016-01-14 16:30:04,849 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/history.txt.appattempt_1452789004804_0001_000000, maxErrors=10
2016-01-14 16:30:04,850 INFO  recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-01-14 16:30:04,850 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1452789004804_0001_000000, appSubmitTime=1452789004835, launchTime=1452789004835
2016-01-14 16:30:04,857 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-01-14 16:30:04,860 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-01-14 16:30:04,861 INFO  client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-187-41-40.ec2.internal/10.187.41.40:51738
2016-01-14 16:30:04,862 INFO  recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-01-14 16:30:04,865 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-01-14 16:30:04,865 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-01-14 16:30:04,867 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1452789004804_0001_000000, startTime=1452789004867
2016-01-14 16:30:04,867 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-01-14 16:30:04,867 INFO  app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-01-14 16:30:04,931 INFO  client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-01-14 16:30:04,931 INFO  client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-01-14 16:30:04,931 INFO  client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001, dagName=[033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,950 INFO  app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04,953 INFO  app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1452789004804_0001_1, filePath=/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/dag_1452789004804_0001_1.dot
2016-01-14 16:30:04,955 INFO  app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001/localmode-log-dir/dag_1452789004804_0001_1-tez-dag.pb.txt
2016-01-14 16:30:04,998 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789004804_0001_1
2016-01-14 16:30:04,998 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:04,998 INFO  app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:04 Running Dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,013 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1452789004804_0001_1, submitTime=1452789004950
2016-01-14 16:30:05,013 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-01-14 16:30:05,014 INFO  impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,015 INFO  impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-01-14 16:30:05,015 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-01-14 16:30:05,015 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1452789004804_0001_1, initTime=1452789005013
2016-01-14 16:30:05,016 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-01-14 16:30:05,017 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_STARTED]: dagID=dag_1452789004804_0001_1, startTime=1452789005016
2016-01-14 16:30:05,017 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-01-14 16:30:05,017 INFO  impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-01-14 16:30:05,017 INFO  client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001, dagName=[033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short
2016-01-14 16:30:05,017 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E3B0FB85E18E4F9BA18C85D043EA8509 : {DA81235C682A4596919C8AEC404F7E66={InputName=DA81235C682A4596919C8AEC404F7E66}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: DA81235C682A4596919C8AEC404F7E66, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,019 INFO  impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]. Starting root input initializers: 1
2016-01-14 16:30:05,020 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from NEW to INITIALIZING due to event V_INIT
2016-01-14 16:30:05,021 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: DA81235C682A4596919C8AEC404F7E66 on vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,021 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,022 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-01-14 16:30:05,023 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input DA81235C682A4596919C8AEC404F7E66 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-01-14 16:30:05,023 INFO  common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-01-14 16:30:05,023 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-01-14 16:30:05,026 INFO  hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 404
2016-01-14 16:30:05,026 INFO  dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: DA81235C682A4596919C8AEC404F7E66 on vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,028 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,028 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,030 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,030 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,031 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, vertexId=vertex_1452789004804_0001_1_01, initRequestedTime=1452789005021, initedTime=1452789005030, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,031 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,032 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:05,032 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {DA81235C682A4596919C8AEC404F7E66=forAllWorkUnits=true, update=[1]}
2016-01-14 16:30:05,033 INFO  impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] parallelism set to 1
2016-01-14 16:30:05,033 INFO  impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-01-14 16:30:05,034 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_00, eventCount=1
2016-01-14 16:30:05,035 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,035 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, vertexId=vertex_1452789004804_0001_1_00, initRequestedTime=1452789005019, initedTime=1452789005034, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-01-14 16:30:05,036 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] with distanceFromRoot: 0
2016-01-14 16:30:05,036 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-01-14 16:30:05,036 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean] submitted tez dag to app master: application_1452789004804_0001, with dag id: dag_1452789004804_0001_1
2016-01-14 16:30:05,036 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,040 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,041 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,041 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,041 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,042 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A016A996D30D455BA1D753AC00B9C434, vertexId=vertex_1452789004804_0001_1_02, initRequestedTime=1452789005036, initedTime=1452789005041, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,043 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,043 INFO  impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,043 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,044 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,045 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_00, startRequestedTime=1452789005043, startedTime=1452789005043
2016-01-14 16:30:05,045 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,045 INFO  impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,053 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-01-14 16:30:05,054 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO  impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,054 INFO  impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-01-14 16:30:05,055 INFO  impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,057 INFO  committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-01-14 16:30:05,057 INFO  impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=B4B02E8B10CD404AAC3C9D5E7208B4A4, vertexId=vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,059 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, vertexId=vertex_1452789004804_0001_1_03, initRequestedTime=1452789005045, initedTime=1452789005059, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-01-14 16:30:05,059 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from NEW to INITED due to event V_INIT
2016-01-14 16:30:05,059 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-01-14 16:30:05,060 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,061 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskId=task_1452789004804_0001_1_00_000000, scheduledTime=1452789005060, launchTime=1452789005060
2016-01-14 16:30:05,061 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,062 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_00 for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,062 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] with distanceFromRoot: 1
2016-01-14 16:30:05,063 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,063 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E3B0FB85E18E4F9BA18C85D043EA8509 in vertex: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,064 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: D2A0861392DE4D708D767DEDBAB90939 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,064 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,064 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_01, startRequestedTime=1452789005063, startedTime=1452789005063
2016-01-14 16:30:05,065 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,065 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: E3B0FB85E18E4F9BA18C85D043EA8509, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,066 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,066 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_01 for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,066 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] with distanceFromRoot: 2
2016-01-14 16:30:05,067 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,067 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D2A0861392DE4D708D767DEDBAB90939 in vertex: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,068 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A016A996D30D455BA1D753AC00B9C434 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,068 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,069 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_02, startRequestedTime=1452789005067, startedTime=1452789005067
2016-01-14 16:30:05,069 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,069 INFO  impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1452789004804_0001_1_02 for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] numStartedSources: 1 numSources: 1
2016-01-14 16:30:05,069 INFO  impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] with distanceFromRoot: 3
2016-01-14 16:30:05,070 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A016A996D30D455BA1D753AC00B9C434 in vertex: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,070 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: AE7FFF9183E44A5CA31F76C68F5532F6 with 1 source tasks and 1 pending tasks
2016-01-14 16:30:05,071 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-01-14 16:30:05,071 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1452789004804_0001_1_03, startRequestedTime=1452789005070, startedTime=1452789005070
2016-01-14 16:30:05,071 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from INITED to RUNNING due to event V_START
2016-01-14 16:30:05,072 INFO  node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-01-14 16:30:05,072 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-01-14 16:30:05,073 INFO  container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-01-14 16:30:05,073 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000001
2016-01-14 16:30:05,074 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,076 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789004804_0001_00_000001
2016-01-14 16:30:05,076 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:05,079 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,079 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,080 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000001, launchTime=1452789005078
2016-01-14 16:30:05,080 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,081 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_00_000000_0] to container: [container_1452789004804_0001_00_000001]
2016-01-14 16:30:05,082 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,082 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_00_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000001] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,083 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, startTime=1452789005082, containerId=container_1452789004804_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000001/teamcity, completedLogs=
2016-01-14 16:30:05,083 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,083 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,083 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,084 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000001 given task: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,084 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,085 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,085 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,085 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,085 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,086 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,086 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,086 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,086 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: E3B0FB85E18E4F9BA18C85D043EA8509, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,086 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,088 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,088 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,088 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,090 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_00_000000_0 sent events: (0-1)
2016-01-14 16:30:05,090 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DA81235C682A4596919C8AEC404F7E66, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-01-14 16:30:05,091 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,094 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,098 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,098 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,099 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,098 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,100 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,100 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,103 INFO  input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-01-14 16:30:05,103 INFO  input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,104 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,105 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,105 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,108 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,109 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, ordinal: 0
2016-01-14 16:30:05,109 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, mem on start (mb), free: 60, total: 516, max: 672
2016-01-14 16:30:05,109 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,110 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,110 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,110 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,110 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,111 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-01-14 16:30:05,111 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:05,111 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,112 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,112 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-01-14 16:30:05,112 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DA81235C682A4596919C8AEC404F7E66, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-01-14 16:30:05,112 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D2A0861392DE4D708D767DEDBAB90939, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-01-14 16:30:05,113 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,113 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,113 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:05,113 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,114 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DA81235C682A4596919C8AEC404F7E66 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,114 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,115 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,115 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,115 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,116 INFO  input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-01-14 16:30:05,116 INFO  input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@6c03cac5
2016-01-14 16:30:05,118 INFO  lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@6c03cac5
2016-01-14 16:30:05,118 INFO  input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-01-14 16:30:05,118 INFO  input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-01-14 16:30:05,120 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:05,121 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"] streamed: true, id: DA81235C682A4596919C8AEC404F7E66
2016-01-14 16:30:05,121 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,122 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,122 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@6796ffe0; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@6a790041; comparator=cascading.tuple.hadoop.util.TupleComparator@2dffe203; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,123 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,123 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,123 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,147 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-01-14 16:30:05,147 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-01-14 16:30:05,147 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-01-14 16:30:05,147 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-01-14 16:30:05,148 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-01-14 16:30:05,149 INFO  element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/classes/test/data/critics.txt"]
2016-01-14 16:30:05,149 INFO  input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-01-14 16:30:05,149 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,153 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,153 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E3B0FB85E18E4F9BA18C85D043EA8509, mem on close (mb), free: 162, total: 517, max: 672
2016-01-14 16:30:05,153 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,154 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,154 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 1676; bufvoid = 104857600
2016-01-14 16:30:05,154 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214260(104857040); length = 137/6553600
2016-01-14 16:30:05,158 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,162 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,163 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_00, eventCount=1
2016-01-14 16:30:05,163 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-01-14 16:30:05,163 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskAttemptId=attempt_1452789004804_0001_1_00_000000_0, startTime=1452789005082, finishTime=1452789005163, timeTaken=81, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:05,164 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:05,163 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,164 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,164 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,164 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,165 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05,165 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,165 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,165 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, taskId=task_1452789004804_0001_1_00_000000, startTime=1452789005082, finishTime=1452789005165, timeTaken=83, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-01-14 16:30:05,166 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,166 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] : 1
2016-01-14 16:30:05,166 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,168 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,168 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=E3B0FB85E18E4F9BA18C85D043EA8509, vertexId=vertex_1452789004804_0001_1_00, initRequestedTime=1452789005019, initedTime=1452789005034, startRequestedTime=1452789005043, startedTime=1452789005043, finishTime=1452789005166, timeTaken=123, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=1274, FILE_BYTES_WRITTEN=1121, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005149, Process_Duration=4, Process_End_Time=1452789005153, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1452789005082, firstTasksToStart=[ task_1452789004804_0001_1_00_000000 ], lastTaskFinishTime=1452789005163, lastTasksToFinish=[ task_1452789004804_0001_1_00_000000 ], minTaskDuration=81, maxTaskDuration=81, avgTaskDuration=81.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_00_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,168 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,168 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,168 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] attempt: attempt_1452789004804_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,169 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,169 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,169 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000001, stoppedTime=1452789005169, exitStatus=0
2016-01-14 16:30:05,169 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,170 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,170 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,171 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskId=task_1452789004804_0001_1_01_000000, scheduledTime=1452789005170, launchTime=1452789005170
2016-01-14 16:30:05,171 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,171 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,172 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,172 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: D2A0861392DE4D708D767DEDBAB90939, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,172 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,172 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000001. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,173 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,173 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,174 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000002
2016-01-14 16:30:05,174 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,176 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789004804_0001_00_000002
2016-01-14 16:30:05,176 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-01-14 16:30:05,177 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000001 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,177 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,178 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,178 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000001: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,178 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000001, Exiting
2016-01-14 16:30:05,178 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,178 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000001
2016-01-14 16:30:05,178 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000002, launchTime=1452789005177
2016-01-14 16:30:05,180 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,181 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,181 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000001 completed successfully
2016-01-14 16:30:05,181 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_01_000000_0] to container: [container_1452789004804_0001_00_000002]
2016-01-14 16:30:05,182 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,182 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000002 given task: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,182 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,182 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_01_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000002] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,183 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,183 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,183 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, startTime=1452789005182, containerId=container_1452789004804_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000002/teamcity, completedLogs=
2016-01-14 16:30:05,184 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,184 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,183 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,185 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,185 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,185 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,185 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,185 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: D2A0861392DE4D708D767DEDBAB90939, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,185 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,186 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,186 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,186 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,187 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,187 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,188 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,189 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,191 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_01_000000_0 sent events: (0-1)
2016-01-14 16:30:05,192 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E3B0FB85E18E4F9BA18C85D043EA8509, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,192 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,192 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,193 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,191 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,192 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,193 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,195 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,195 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,196 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,196 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,200 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, ordinal: 1
2016-01-14 16:30:05,200 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, mem on start (mb), free: 155, total: 517, max: 672
2016-01-14 16:30:05,200 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,200 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,201 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,201 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,201 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,201 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,202 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,202 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,202 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:05,202 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A016A996D30D455BA1D753AC00B9C434, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:05,203 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E3B0FB85E18E4F9BA18C85D043EA8509, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:05,203 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,203 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E3B0FB85E18E4F9BA18C85D043EA8509 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,203 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,203 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,204 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,205 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,205 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,206 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E3B0FB85E18E4F9BA18C85D043EA8509 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,206 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:05,206 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,206 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,207 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,207 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,208 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,209 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_00_000000_0_10039, runDuration: 60175, ]
2016-01-14 16:30:05,210 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-01-14 16:30:05,211 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_00_000000_0_10039] to DISK_DIRECT, CompressedSize=1089, DecompressedSize=1085,EndTime=1452789005211, TimeTaken=0, Rate=0.00 MB/s
2016-01-14 16:30:05,211 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,211 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 276E68A215354EAC919589E262B6FC62
2016-01-14 16:30:05,211 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:05,211 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,211 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,212 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@25d6868; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@4271850e; comparator=cascading.tuple.hadoop.util.TupleComparator@2ad67c36; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,212 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,212 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,212 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,211 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [E3B0FB85E18E4F9BA18C85D043EA8509] #10 in 1ms
2016-01-14 16:30:05,214 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E3B0FB85E18E4F9BA18C85D043EA8509] #10, status:false, isInterrupted:false
2016-01-14 16:30:05,221 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:05,221 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:05,221 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:05,222 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:05,222 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:05,221 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,223 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 1089 bytes from disk
2016-01-14 16:30:05,223 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,223 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,224 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-01-14 16:30:05,224 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,226 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,226 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 276E68A215354EAC919589E262B6FC62, for 1 inputs
2016-01-14 16:30:05,226 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:05,227 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-01-14 16:30:05,227 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,228 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@5c2b7b8b; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@2abc11f3
2016-01-14 16:30:05,254 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,254 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2A0861392DE4D708D767DEDBAB90939, mem on close (mb), free: 89, total: 517, max: 672
2016-01-14 16:30:05,254 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E3B0FB85E18E4F9BA18C85D043EA8509
2016-01-14 16:30:05,254 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,254 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,254 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,254 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,256 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05,256 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,256 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,256 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 13104; bufvoid = 66060288
2016-01-14 16:30:05,256 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514376(66057504); length = 693/4128768
2016-01-14 16:30:05,264 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,267 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,268 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3]]
2016-01-14 16:30:05,268 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:05,269 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,269 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_01, eventCount=1
2016-01-14 16:30:05,269 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,270 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,270 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskAttemptId=attempt_1452789004804_0001_1_01_000000_0, startTime=1452789005182, finishTime=1452789005269, timeTaken=87, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3
2016-01-14 16:30:05,270 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,270 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05,271 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, taskId=task_1452789004804_0001_1_01_000000, startTime=1452789005182, finishTime=1452789005270, timeTaken=88, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_01_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3
2016-01-14 16:30:05,271 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,271 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] : 1
2016-01-14 16:30:05,272 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,271 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,271 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,273 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=D2A0861392DE4D708D767DEDBAB90939, vertexId=vertex_1452789004804_0001_1_01, initRequestedTime=1452789005021, initedTime=1452789005030, startRequestedTime=1452789005063, startedTime=1452789005063, finishTime=1452789005272, timeTaken=209, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=1153, FILE_BYTES_WRITTEN=7551, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=542638080, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005227, Process_Duration=26, Process_End_Time=1452789005253, Read_Duration=3, Tuples_Read=35, Tuples_Written=174, Write_Duration=3, vertexStats=firstTaskStartTime=1452789005182, firstTasksToStart=[ task_1452789004804_0001_1_01_000000 ], lastTaskFinishTime=1452789005269, lastTasksToFinish=[ task_1452789004804_0001_1_01_000000 ], minTaskDuration=87, maxTaskDuration=87, avgTaskDuration=87.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_01_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,273 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,273 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] attempt: attempt_1452789004804_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,273 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,273 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,273 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,274 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,274 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000002, stoppedTime=1452789005274, exitStatus=0
2016-01-14 16:30:05,274 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,274 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,275 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskId=task_1452789004804_0001_1_02_000000, scheduledTime=1452789005275, launchTime=1452789005275
2016-01-14 16:30:05,275 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,275 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,275 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,275 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,275 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000002. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,276 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A016A996D30D455BA1D753AC00B9C434, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,276 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,276 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,276 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,277 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000003
2016-01-14 16:30:05,278 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,279 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789004804_0001_00_000003
2016-01-14 16:30:05,279 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:05,280 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000002 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,280 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,280 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000002, Exiting
2016-01-14 16:30:05,280 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,281 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000002
2016-01-14 16:30:05,282 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,282 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000002 completed successfully
2016-01-14 16:30:05,284 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000002 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,284 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,284 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,285 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000003, launchTime=1452789005282
2016-01-14 16:30:05,285 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,286 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_02_000000_0] to container: [container_1452789004804_0001_00_000003]
2016-01-14 16:30:05,286 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,287 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_02_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000003] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,287 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, startTime=1452789005287, containerId=container_1452789004804_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000003/teamcity, completedLogs=
2016-01-14 16:30:05,287 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,287 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,287 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,288 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000003 given task: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,288 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,289 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,289 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,289 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,289 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,289 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,289 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: A016A996D30D455BA1D753AC00B9C434, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-01-14 16:30:05,289 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,290 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,290 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,291 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,291 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_02_000000_0 sent events: (0-1)
2016-01-14 16:30:05,291 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,293 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=AE7FFF9183E44A5CA31F76C68F5532F6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-01-14 16:30:05,294 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,294 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,294 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D2A0861392DE4D708D767DEDBAB90939, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,294 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,297 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,297 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,298 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,299 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,299 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,299 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,300 INFO  impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-01-14 16:30:05,300 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,300 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,307 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A016A996D30D455BA1D753AC00B9C434, ordinal: 2
2016-01-14 16:30:05,307 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A016A996D30D455BA1D753AC00B9C434, mem on start (mb), free: 84, total: 517, max: 672
2016-01-14 16:30:05,307 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,307 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,307 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,307 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,307 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,308 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,308 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-01-14 16:30:05,308 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,308 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 739036352, TotalRequestedScaled: 3.69518176E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:1.05
2016-01-14 16:30:05,308 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D2A0861392DE4D708D767DEDBAB90939, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=405125774
2016-01-14 16:30:05,308 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, AE7FFF9183E44A5CA31F76C68F5532F6, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=66985083
2016-01-14 16:30:05,309 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,309 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D2A0861392DE4D708D767DEDBAB90939 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,309 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,309 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,310 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,310 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,311 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,311 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D2A0861392DE4D708D767DEDBAB90939 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,312 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=405125774. Updated to: ShuffleMem=405125774, postMergeMem: 0
2016-01-14 16:30:05,312 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=405125774, maxSingleShuffleLimit=101281440, mergeThreshold=364613184, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,312 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,312 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,313 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,313 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,316 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_01_000000_0_10041, runDuration: 76697, ]
2016-01-14 16:30:05,319 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_01_000000_0_10041] to DISK_DIRECT, CompressedSize=7519, DecompressedSize=7515,EndTime=1452789005319, TimeTaken=1, Rate=7.17 MB/s
2016-01-14 16:30:05,319 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,319 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-01-14 16:30:05,320 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [D2A0861392DE4D708D767DEDBAB90939] #11 in 3ms
2016-01-14 16:30:05,320 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-01-14 16:30:05,320 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D2A0861392DE4D708D767DEDBAB90939] #11, status:false, isInterrupted:false
2016-01-14 16:30:05,321 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,321 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: ADBFE874A1E34C0892F9528F790D6501
2016-01-14 16:30:05,321 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 7519 bytes from disk
2016-01-14 16:30:05,321 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,321 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,321 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,322 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-01-14 16:30:05,322 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,322 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,322 INFO  element.TezGroupGate (TezGroupGate.java:prepare(96)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,323 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TupleSerializer@59cf288; valueSerializer=cascading.tuple.hadoop.io.TupleSerializer@2f6c199; comparator=cascading.tuple.hadoop.util.TupleComparator@16202554; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-01-14 16:30:05,323 INFO  impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-01-14 16:30:05,323 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-01-14 16:30:05,323 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,341 INFO  dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 16515068(66060272)
2016-01-14 16:30:05,341 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 63
2016-01-14 16:30:05,341 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 52848232
2016-01-14 16:30:05,341 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 66060288
2016-01-14 16:30:05,341 INFO  dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 16515068; length = 4128768
2016-01-14 16:30:05,342 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] ADBFE874A1E34C0892F9528F790D6501, for 1 inputs
2016-01-14 16:30:05,342 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A016A996D30D455BA1D753AC00B9C434, all 1 inputs ready in: 00:00:00.000
2016-01-14 16:30:05,343 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@405e2da9; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@6707d342
2016-01-14 16:30:05,364 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,364 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A016A996D30D455BA1D753AC00B9C434, mem on close (mb), free: 186, total: 503, max: 672
2016-01-14 16:30:05,364 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D2A0861392DE4D708D767DEDBAB90939
2016-01-14 16:30:05,365 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,365 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,365 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,365 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,366 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05,366 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-01-14 16:30:05,366 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-01-14 16:30:05,366 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 7074; bufvoid = 66060288
2016-01-14 16:30:05,367 INFO  dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 16515068(66060272); kvend = 16514724(66058896); length = 345/4128768
2016-01-14 16:30:05,370 INFO  dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,374 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,375 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1452789004804_0001_1_02, eventCount=1
2016-01-14 16:30:05,375 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 49 [[File System Counters FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-01-14 16:30:05,375 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,376 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:05,376 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,376 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,376 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskAttemptId=attempt_1452789004804_0001_1_02_000000_0, startTime=1452789005287, finishTime=1452789005375, timeTaken=88, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 16:30:05,376 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,377 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,377 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05,377 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,378 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, taskId=task_1452789004804_0001_1_02_000000, startTime=1452789005287, finishTime=1452789005377, timeTaken=90, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_02_000000_0, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-01-14 16:30:05,378 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,378 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] : 1
2016-01-14 16:30:05,378 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,379 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,380 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=A016A996D30D455BA1D753AC00B9C434, vertexId=vertex_1452789004804_0001_1_02, initRequestedTime=1452789005036, initedTime=1452789005041, startRequestedTime=1452789005067, startedTime=1452789005067, finishTime=1452789005378, timeTaken=311, status=SUCCEEDED, diagnostics=, counters=Counters: 49, File System Counters, FILE_BYTES_READ=7583, FILE_BYTES_WRITTEN=5159, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=86, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=10, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005342, Process_Duration=22, Process_End_Time=1452789005364, Read_Duration=10, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1452789005287, firstTasksToStart=[ task_1452789004804_0001_1_02_000000 ], lastTaskFinishTime=1452789005375, lastTasksToFinish=[ task_1452789004804_0001_1_02_000000 ], minTaskDuration=88, maxTaskDuration=88, avgTaskDuration=88.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_02_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,380 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,380 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,380 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000003, stoppedTime=1452789005380, exitStatus=0
2016-01-14 16:30:05,380 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,380 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05,381 INFO  impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] attempt: attempt_1452789004804_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-01-14 16:30:05,381 INFO  vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AE7FFF9183E44A5CA31F76C68F5532F6
2016-01-14 16:30:05,381 INFO  impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,381 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,382 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000003. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,382 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_STARTED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskId=task_1452789004804_0001_1_03_000000, scheduledTime=1452789005382, launchTime=1452789005382
2016-01-14 16:30:05,382 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-01-14 16:30:05,382 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-01-14 16:30:05,383 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,383 INFO  impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,383 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: AE7FFF9183E44A5CA31F76C68F5532F6, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:05,383 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-01-14 16:30:05,384 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-01-14 16:30:05,385 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1452789004804_0001_00_000004
2016-01-14 16:30:05,385 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-01-14 16:30:05,386 INFO  task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier:  null, container_1452789004804_0001_00_000004
2016-01-14 16:30:05,387 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000003 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,387 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,387 WARN  impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-01-14 16:30:05,387 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000003, Exiting
2016-01-14 16:30:05,388 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,388 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000003
2016-01-14 16:30:05,389 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1452789004804_0001_00_000004, launchTime=1452789005389
2016-01-14 16:30:05,391 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1452789004804_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-01-14 16:30:05,391 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1452789004804_0001_00_000003 completed successfully
2016-01-14 16:30:05,389 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-01-14 16:30:05,392 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000003 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-01-14 16:30:05,389 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-01-14 16:30:05,392 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,393 INFO  container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1452789004804_0001_1_03_000000_0] to container: [container_1452789004804_0001_00_000004]
2016-01-14 16:30:05,393 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-01-14 16:30:05,394 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1452789004804_0001_00_000004 given task: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,394 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,394 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1452789004804_0001_1_03_000000_0] started. Is using containerId: [container_1452789004804_0001_00_000004] on NM: [127.0.0.1:0]
2016-01-14 16:30:05,394 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,395 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,395 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-01-14 16:30:05,395 INFO  task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-01-14 16:30:05,395 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-01-14 16:30:05,395 INFO  task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-01-14 16:30:05,395 INFO  metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) -  Using ResourceCalculatorProcessTree : [ ]
2016-01-14 16:30:05,395 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: AE7FFF9183E44A5CA31F76C68F5532F6, VertexParallelism: 1, TaskAttemptID:attempt_1452789004804_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-01-14 16:30:05,396 INFO  resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-01-14 16:30:05,395 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-01-14 16:30:05,395 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, startTime=1452789005394, containerId=container_1452789004804_0001_00_000004, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1452789004804_0001_00_000004/teamcity, completedLogs=
2016-01-14 16:30:05,396 INFO  task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,396 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-01-14 16:30:05,396 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,397 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A016A996D30D455BA1D753AC00B9C434, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-01-14 16:30:05,398 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-01-14 16:30:05,396 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-01-14 16:30:05,399 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,401 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,401 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-01-14 16:30:05,402 INFO  impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1452789004804_0001_1_03_000000_0 sent events: (0-1)
2016-01-14 16:30:05,402 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B4B02E8B10CD404AAC3C9D5E7208B4A4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-01-14 16:30:05,403 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-01-14 16:30:05,403 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,403 INFO  output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-01-14 16:30:05,403 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,405 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.0.4
2016-01-14 16:30:05,405 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-01-14 16:30:05,405 INFO  orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-01-14 16:30:05,405 INFO  resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,409 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,417 INFO  tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, ordinal: 3
2016-01-14 16:30:05,417 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, mem on start (mb), free: 179, total: 503, max: 672
2016-01-14 16:30:05,417 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-01-14 16:30:05,418 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-01-14 16:30:05,418 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-01-14 16:30:05,421 INFO  output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-01-14 16:30:05,422 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,422 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-01-14 16:30:05,422 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-01-14 16:30:05,422 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-01-14 16:30:05,423 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-01-14 16:30:05,423 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-01-14 16:30:05,423 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-01-14 16:30:05,423 INFO  resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-01-14 16:30:05,423 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B4B02E8B10CD404AAC3C9D5E7208B4A4, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-01-14 16:30:05,423 INFO  resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A016A996D30D455BA1D753AC00B9C434, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-01-14 16:30:05,423 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-01-14 16:30:05,424 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,424 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A016A996D30D455BA1D753AC00B9C434 being auto started by the framework. Subsequent instances will not be auto-started
2016-01-14 16:30:05,424 INFO  common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-01-14 16:30:05,426 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-01-14 16:30:05,427 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A016A996D30D455BA1D753AC00B9C434 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-01-14 16:30:05,427 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-01-14 16:30:05,427 INFO  orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-01-14 16:30:05,427 INFO  orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-01-14 16:30:05,429 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-01-14 16:30:05,429 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-01-14 16:30:05,429 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,430 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-01-14 16:30:05,430 INFO  task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,431 INFO  orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-187-41-40, port: 0, pathComponent: attempt_1452789004804_0001_1_02_000000_0_10044, runDuration: 76941, ]
2016-01-14 16:30:05,433 INFO  graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-01-14 16:30:05,434 INFO  tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 6A1ACBEDEA3040F593F4450750E64F8E
2016-01-14 16:30:05,434 INFO  tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: B4B02E8B10CD404AAC3C9D5E7208B4A4
2016-01-14 16:30:05,434 INFO  element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-01-14 16:30:05,435 INFO  element.TezGroupGate (TezGroupGate.java:prepare(88)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 6A1ACBEDEA3040F593F4450750E64F8E, for 1 inputs
2016-01-14 16:30:05,435 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1452789004804_0001_1_02_000000_0_10044] to DISK_DIRECT, CompressedSize=5127, DecompressedSize=5123,EndTime=1452789005435, TimeTaken=1, Rate=4.89 MB/s
2016-01-14 16:30:05,436 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,436 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-01-14 16:30:05,436 INFO  orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-187-41-40:0 freed by fetcher [A016A996D30D455BA1D753AC00B9C434] #12 in 3ms
2016-01-14 16:30:05,438 INFO  orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A016A996D30D455BA1D753AC00B9C434] #12, status:false, isInterrupted:false
2016-01-14 16:30:05,439 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-01-14 16:30:05,440 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 5127 bytes from disk
2016-01-14 16:30:05,440 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,440 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-01-14 16:30:05,441 INFO  impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-01-14 16:30:05,441 INFO  orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,442 INFO  orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-01-14 16:30:05,441 INFO  tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, all 1 inputs ready in: 00:00:00.006
2016-01-14 16:30:05,445 INFO  common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@244ff83; valueDeserializer=cascading.tuple.hadoop.io.TupleDeserializer@7fe2f29e
2016-01-14 16:30:05,455 INFO  output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-01-14 16:30:05,455 INFO  output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-01-14 16:30:05,455 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,456 INFO  impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1452789004804_0001_1_03_000000_0 given a go for committing the task output.
2016-01-14 16:30:05,457 INFO  task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,457 INFO  tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AE7FFF9183E44A5CA31F76C68F5532F6, mem on close (mb), free: 176, total: 503, max: 672
2016-01-14 16:30:05,458 INFO  orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A016A996D30D455BA1D753AC00B9C434
2016-01-14 16:30:05,459 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-01-14 16:30:05,459 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-01-14 16:30:05,459 INFO  orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-01-14 16:30:05,460 INFO  impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-01-14 16:30:05,460 INFO  impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,462 INFO  task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, fatalErrorOccurred=false
2016-01-14 16:30:05,464 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-01-14 16:30:05,464 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskAttemptId=attempt_1452789004804_0001_1_03_000000_0, startTime=1452789005394, finishTime=1452789005464, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:05,464 INFO  impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1452789004804_0001_1_03_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-01-14 16:30:05,464 INFO  impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05,465 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-01-14 16:30:05,465 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:TASK_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, taskId=task_1452789004804_0001_1_03_000000, startTime=1452789005394, finishTime=1452789005464, timeTaken=70, status=SUCCEEDED, successfulAttemptID=attempt_1452789004804_0001_1_03_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-01-14 16:30:05,465 INFO  impl.TaskImpl (TaskImpl.java:handle(862)) - task_1452789004804_0001_1_03_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-01-14 16:30:05,465 INFO  impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] : 1
2016-01-14 16:30:05,466 INFO  impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,466 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-01-14 16:30:05,466 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-01-14 16:30:05,467 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:VERTEX_FINISHED]: vertexName=AE7FFF9183E44A5CA31F76C68F5532F6, vertexId=vertex_1452789004804_0001_1_03, initRequestedTime=1452789005045, initedTime=1452789005059, startRequestedTime=1452789005070, startedTime=1452789005070, finishTime=1452789005466, timeTaken=396, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1452789005394, firstTasksToStart=[ task_1452789004804_0001_1_03_000000 ], lastTaskFinishTime=1452789005464, lastTasksToFinish=[ task_1452789004804_0001_1_03_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1452789004804_0001_1_03_000000 ], longestDurationTasks=[ task_1452789004804_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-01-14 16:30:05,467 INFO  impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-01-14 16:30:05,467 INFO  runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=5159, FILE_BYTES_WRITTEN=1420, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=527433728, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1452789005442, Process_Duration=15, Process_End_Time=1452789005457, Read_Duration=5, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-01-14 16:30:05,467 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-01-14 16:30:05,468 INFO  impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-01-14 16:30:05,468 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,468 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: 
2016-01-14 16:30:05,468 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,469 INFO  task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,469 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1452789004804_0001_1
2016-01-14 16:30:05,469 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: B4B02E8B10CD404AAC3C9D5E7208B4A4 for vertex: vertex_1452789004804_0001_1_03 [AE7FFF9183E44A5CA31F76C68F5532F6]
2016-01-14 16:30:05,479 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_02 [A016A996D30D455BA1D753AC00B9C434]
2016-01-14 16:30:05,480 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_01 [D2A0861392DE4D708D767DEDBAB90939]
2016-01-14 16:30:05,480 INFO  impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1452789004804_0001_1_00 [E3B0FB85E18E4F9BA18C85D043EA8509]
2016-01-14 16:30:05,483 INFO  recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1452789004804_0001_1, queueSize=0
2016-01-14 16:30:05,484 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-01-14 16:30:05,485 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:DAG_FINISHED]: dagId=dag_1452789004804_0001_1, startTime=1452789005016, finishTime=1452789005480, timeTaken=464, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=15169, FILE_BYTES_WRITTEN=15251, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=111, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, GC_TIME_MILLIS=20, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=2140143616, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=30, MERGE_PHASE_TIME=45, FIRST_EVENT_RECEIVED=17, LAST_EVENT_RECEIVED=17, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5811156021160, Process_Duration=67, Process_End_Time=5811156021227, Read_Duration=19, Tuples_Read=303, Tuples_Written=317, Write_Duration=7, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-01-14 16:30:05,485 INFO  impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1452789004804_0001_1 finished with state: SUCCEEDED
2016-01-14 16:30:05,485 INFO  impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1452789004804_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-01-14 16:30:05,486 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-01-14 16:30:05,487 INFO  history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1452789004804_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1452789004804_0001_00_000004, stoppedTime=1452789005486, exitStatus=0
2016-01-14 16:30:05,487 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-01-14 16:30:05 Completed Dag: dag_1452789004804_0001_1
2016-01-14 16:30:05,489 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1452789004804_0001_1, dagState=SUCCEEDED
2016-01-14 16:30:05,489 INFO  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1452789004804_0001_1_post
2016-01-14 16:30:05,490 WARN  common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-01-14 16:30:05,490 INFO  app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-01-14 16:30:05,491 INFO  launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1452789004804_0001_00_000004. Relying on regular task shutdown for it to end
2016-01-14 16:30:05,491 INFO  container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1452789004804_0001_00_000004 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-01-14 16:30:05,494 INFO  client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1452789004804_0001
2016-01-14 16:30:05,496 INFO  client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-01-14 16:30:05,496 INFO  rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-01-14 16:30:05,496 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-01-14 16:30:05,496 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-01-14 16:30:05,496 INFO  app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-01-14 16:30:05,497 INFO  app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-01-14 16:30:05,497 INFO  history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-01-14 16:30:05,498 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-01-14 16:30:05,498 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-01-14 16:30:05,498 INFO  recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-01-14 16:30:05,498 INFO  recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-01-14 16:30:05,499 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-01-14 16:30:05,499 INFO  impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-01-14 16:30:05,499 INFO  app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1452789004804_0001_00_000004 is valid, but no longer registered, and will be killed
2016-01-14 16:30:05,499 INFO  task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1452789004804_0001_00_000004: 30 ms after starting to poll. TaskInfo: shouldDie: true
2016-01-14 16:30:05,499 INFO  task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1452789004804_0001_00_000004, Exiting
2016-01-14 16:30:05,500 INFO  task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,500 INFO  task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1452789004804_0001_00_000004
2016-01-14 16:30:05,510 INFO  app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/033E240DA4D248B2935CC61F500F33FA/A2282440304B41ACA549929830082BDC/.tez/application_1452789004804_0001
2016-01-14 16:30:05,511 INFO  flow.Flow (BaseFlow.java:logInfo(1433)) - [euclidean]  completed in: 00:00.726
2016-01-14 16:30:05,585 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/ca57560bc31094a7/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short

Standard error

2016-01-14 16:30:00 Running Dag: dag_1452788999139_0001_1
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00 Completed running task attempt: attempt_1452788999139_0001_1_00_000000_0
2016-01-14 16:30:00 Starting to run new task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_01_000000_0
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_02_000000_0
2016-01-14 16:30:01 Starting to run new task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01 Completed running task attempt: attempt_1452788999139_0001_1_03_000000_0
2016-01-14 16:30:01 Completed Dag: dag_1452788999139_0001_1
2016-01-14 16:30:02 Running Dag: dag_1452789002288_0001_1
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_00_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02 Completed running task attempt: attempt_1452789002288_0001_1_01_000000_0
2016-01-14 16:30:02 Starting to run new task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_02_000000_0
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03 Completed running task attempt: attempt_1452789002288_0001_1_03_000000_0
2016-01-14 16:30:03 Completed Dag: dag_1452789002288_0001_1
2016-01-14 16:30:03 Running Dag: dag_1452789003647_0001_1
2016-01-14 16:30:03 Starting to run new task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_00_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_01_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_02_000000_0
2016-01-14 16:30:04 Starting to run new task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04 Completed running task attempt: attempt_1452789003647_0001_1_03_000000_0
2016-01-14 16:30:04 Completed Dag: dag_1452789003647_0001_1
2016-01-14 16:30:04 Running Dag: dag_1452789004804_0001_1
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_00_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_01_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_02_000000_0
2016-01-14 16:30:05 Starting to run new task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05 Completed running task attempt: attempt_1452789004804_0001_1_03_000000_0
2016-01-14 16:30:05 Completed Dag: dag_1452789004804_0001_1
Jan 14, 2016 4:30:05 PM com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair execute
SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@6c0f7339 with executor java.util.concurrent.ThreadPoolExecutor@6a55da9b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
java.util.concurrent.RejectedExecutionException: Task com.google.common.util.concurrent.Futures$6@6c0f7339 rejected from java.util.concurrent.ThreadPoolExecutor@6a55da9b[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 3]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
	at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
	at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:86)
	at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
	at java.util.concurrent.FutureTask.set(FutureTask.java:229)
	at java.util.concurrent.FutureTask.run(FutureTask.java:270)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)