2016-05-18 00:13:09,120 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-18 00:13:09,151 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.DistanceUseCasePlatformTest, with platform: hadoop2-tez
2016-05-18 00:13:09,183 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-18 00:13:09,734 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-18 00:13:10,076 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-18 00:13:10,402 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:13:10,404 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:13:10,526 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: 5EC8B9416E8643F695707F1D9C33B54C
2016-05-18 00:13:10,804 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.164
2016-05-18 00:13:10,848 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.209
2016-05-18 00:13:10,849 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:10,850 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:10,852 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:13:11,028 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt
2016-05-18 00:13:11,233 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-05-18 00:13:11,594 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-18 00:13:11,597 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-18 00:13:11,597 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:11,598 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-05-18 00:13:11,598 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-18 00:13:11,598 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-18 00:13:11,599 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-18 00:13:11,600 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...osite/euclidean/composite
2016-05-18 00:13:11,601 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:13:11,620 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-18 00:13:11,794 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:11,804 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:13:11,804 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:13:11,873 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001 doesn't exist and is created
2016-05-18 00:13:11,960 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:13:11,960 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:11,962 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd
2016-05-18 00:13:12,024 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463530391803_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:12,038 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:12,039 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:12,045 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:13:12,046 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:13:12,046 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:13:12,047 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:13:12,061 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:13:12,061 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:13:12,061 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:12,073 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:13:12,103 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:13:12,105 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:13:12,107 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:13:12,108 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:13:12,109 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:13:12,109 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:13:12,111 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:13:12,112 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:13:12,114 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:12,114 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:12,147 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:13:12,152 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:13:12,161 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:13:12,162 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:13:12,162 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:12,172 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:13:12,174 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:13:12,174 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:13:12,178 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-log-dir
2016-05-18 00:13:12,178 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-log-dir/history.txt.appattempt_1463530391803_0001_000000, maxErrors=10
2016-05-18 00:13:12,178 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:13:12,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463530391803_0001_000000, appSubmitTime=1463530392003, launchTime=1463530392022
2016-05-18 00:13:12,190 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:13:12,248 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:33298
2016-05-18 00:13:12,251 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463530391803
2016-05-18 00:13:12,262 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-18 00:13:12,262 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-18 00:13:12,262 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:12,269 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463530391803_0001_000000, startTime=1463530392268
2016-05-18 00:13:12,269 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:13:12,362 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:13:12,379 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:13:12,379 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463530391803_0001, dagName=[3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite
2016-05-18 00:13:12,584 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite
2016-05-18 00:13:12,592 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:13:12,655 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463530391803_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-log-dir/dag_1463530391803_0001_1.dot
2016-05-18 00:13:12,660 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-log-dir/dag_1463530391803_0001_1-tez-dag.pb.txt
2016-05-18 00:13:12,709 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530391803_0001_1
2016-05-18 00:13:12,709 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:12,710 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite
2016-05-18 00:13:12 Running Dag: dag_1463530391803_0001_1
2016-05-18 00:13:12,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463530391803_0001_1, submitTime=1463530392584
2016-05-18 00:13:12,793 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001/tez-conf.pb" } size: 36897 timestamp: 1463530391000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:12,793 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,793 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:12,795 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001/tez-conf.pb" } size: 36897 timestamp: 1463530391000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:12,795 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:12,797 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001/tez-conf.pb" } size: 36897 timestamp: 1463530391000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:12,797 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:12,799 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001/tez-conf.pb" } size: 36897 timestamp: 1463530391000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:12,799 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:12,800 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:12,808 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:13:12,810 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463530391803_0001_1, initTime=1463530392810
2016-05-18 00:13:12,811 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530391803_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:13:12,814 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:13:12,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:DAG_STARTED]: dagID=dag_1463530391803_0001_1, startTime=1463530392816
2016-05-18 00:13:12,816 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530391803_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:13:12,816 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CA8D8362F2F942B29BF2CD6A11E8A1AF : {24DC3C94BEAE464BAE95E714DB083EE0={InputName=24DC3C94BEAE464BAE95E714DB083EE0}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:13:12,818 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 24DC3C94BEAE464BAE95E714DB083EE0, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:13:12,818 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,826 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463530391803_0001, dagName=[3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite
2016-05-18 00:13:12,830 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,830 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,832 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,839 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 24DC3C94BEAE464BAE95E714DB083EE0 on vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,851 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:13:12,852 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:12,872 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463530391803_0001, with dag id: dag_1463530391803_0001_1
2016-05-18 00:13:12,877 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 24DC3C94BEAE464BAE95E714DB083EE0 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:13:12,904 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:12,906 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:12,912 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:13:12,913 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 828 Min splitLength: 52428800 New desired splits: 1 Total length: 828 Original splits: 2
2016-05-18 00:13:12,914 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 828 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 828 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:13:12,915 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:13:12,925 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:12,930 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 24DC3C94BEAE464BAE95E714DB083EE0 on vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:12,931 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530391803_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:12,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B8EA621789834CBA90DF275648F8C275, vertexId=vertex_1463530391803_0001_1_01, initRequestedTime=1463530392852, initedTime=1463530392933, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:12,936 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:12,945 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:12,948 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:12,955 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {24DC3C94BEAE464BAE95E714DB083EE0=forAllWorkUnits=true, update=[1]}
2016-05-18 00:13:12,956 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] parallelism set to 1
2016-05-18 00:13:12,956 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:12,991 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:12,991 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:12,992 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:12,993 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530391803_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:12,994 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, vertexId=vertex_1463530391803_0001_1_02, initRequestedTime=1463530392945, initedTime=1463530392994, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:12,995 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:12,998 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, vertexId=vertex_1463530391803_0001_1_00, initRequestedTime=1463530392816, initedTime=1463530392996, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:13:12,998 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] with distanceFromRoot: 0
2016-05-18 00:13:12,998 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:13:12,999 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:13,034 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:13,034 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:13,036 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:13,037 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530391803_0001_1_03, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:13,037 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336], numAdditionalOutputs=1
2016-05-18 00:13:13,038 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=A3F040E13A344444A10F613545D9C067, vertex=vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:13:13,071 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for EDFB22285A7E4CB5A5C46FB0F9083336:A3F040E13A344444A10F613545D9C067 using oldmapred API
2016-05-18 00:13:13,074 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=A3F040E13A344444A10F613545D9C067, vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:13:13,088 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, vertexId=vertex_1463530391803_0001_1_03, initRequestedTime=1463530392999, initedTime=1463530393037, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:13,089 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:13,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530391803_0001_1_00, startRequestedTime=1463530392852, startedTime=1463530393090
2016-05-18 00:13:13,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530391803_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:13:13,093 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:13,095 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CA8D8362F2F942B29BF2CD6A11E8A1AF
2016-05-18 00:13:13,095 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:13,100 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530391803_0001_1_00 for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] numStartedSources: 1 numSources: 1
2016-05-18 00:13:13,101 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] with distanceFromRoot: 1
2016-05-18 00:13:13,106 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_STARTED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, taskId=task_1463530391803_0001_1_00_000000, scheduledTime=1463530393104, launchTime=1463530393104
2016-05-18 00:13:13,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530391803_0001_1_01, startRequestedTime=1463530393100, startedTime=1463530393138
2016-05-18 00:13:13,140 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:13,143 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:13,147 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530391803_0001_1_01 for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] numStartedSources: 1 numSources: 1
2016-05-18 00:13:13,149 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] with distanceFromRoot: 2
2016-05-18 00:13:13,149 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B8EA621789834CBA90DF275648F8C275 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:13,156 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CA8D8362F2F942B29BF2CD6A11E8A1AF in vertex: B8EA621789834CBA90DF275648F8C275 numBipartiteSourceTasks: 1
2016-05-18 00:13:13,162 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530391803_0001_1_02, startRequestedTime=1463530393147, startedTime=1463530393159
2016-05-18 00:13:13,162 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:13,162 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530391803_0001_1_02 for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] numStartedSources: 1 numSources: 1
2016-05-18 00:13:13,162 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] with distanceFromRoot: 3
2016-05-18 00:13:13,163 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530391803_0001_1_03, startRequestedTime=1463530393162, startedTime=1463530393163
2016-05-18 00:13:13,163 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:13,164 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EFFA0841F266450AB4F0E09C4C3D6002 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:13,168 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EDFB22285A7E4CB5A5C46FB0F9083336 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:13,169 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B8EA621789834CBA90DF275648F8C275 in vertex: EFFA0841F266450AB4F0E09C4C3D6002 numBipartiteSourceTasks: 1
2016-05-18 00:13:13,172 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EFFA0841F266450AB4F0E09C4C3D6002 in vertex: EDFB22285A7E4CB5A5C46FB0F9083336 numBipartiteSourceTasks: 1
2016-05-18 00:13:13,218 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:13:13,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530391803_0001_00_000001, launchTime=1463530393233
2016-05-18 00:13:13,238 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000001
2016-05-18 00:13:13,242 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,244 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530391803_0001_1_00_000000_0] started. Is using containerId: [container_1463530391803_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:13:13 Starting to run new task attempt: attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,244 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,245 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:13,246 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:13,250 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0, startTime=1463530393242, containerId=container_1463530391803_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:13:13,255 INFO counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-18 00:13:13,258 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, VertexName: CA8D8362F2F942B29BF2CD6A11E8A1AF, VertexParallelism: 1, TaskAttemptID:attempt_1463530391803_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=24DC3C94BEAE464BAE95E714DB083EE0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B8EA621789834CBA90DF275648F8C275, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:13,260 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:13,264 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,275 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463530391803_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:13:13,277 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530391803_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:13:13,291 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:13,315 INFO input.MRInput (MRInput.java:initialize(440)) - 24DC3C94BEAE464BAE95E714DB083EE0 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:13:13,316 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 24DC3C94BEAE464BAE95E714DB083EE0
2016-05-18 00:13:13,332 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CA8D8362F2F942B29BF2CD6A11E8A1AF, ordinal: 0
2016-05-18 00:13:13,333 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA8D8362F2F942B29BF2CD6A11E8A1AF, mem on start (mb), free: 174, total: 281, max: 672
2016-05-18 00:13:13,333 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:13,333 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:13,333 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:13,333 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:13,334 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[24DC3C94BEAE464BAE95E714DB083EE0:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B8EA621789834CBA90DF275648F8C275:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:13,338 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:13,346 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:13,346 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:13:13,347 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[24DC3C94BEAE464BAE95E714DB083EE0:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B8EA621789834CBA90DF275648F8C275:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:13,347 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:13,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 24DC3C94BEAE464BAE95E714DB083EE0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:13,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 24DC3C94BEAE464BAE95E714DB083EE0
2016-05-18 00:13:13,349 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:13,349 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:13,349 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:13,351 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,384 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:0+414, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:414+414], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=828}
2016-05-18 00:13:13,385 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 24DC3C94BEAE464BAE95E714DB083EE0 initialized RecordReader from event
2016-05-18 00:13:13,409 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:13,448 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"] streamed: true, id: 24DC3C94BEAE464BAE95E714DB083EE0
2016-05-18 00:13:13,448 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]], id: 8B241D0ED4E54E05813803A2240216F1
2016-05-18 00:13:13,448 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 8B241D0ED4E54E05813803A2240216F1
2016-05-18 00:13:13,501 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B8EA621789834CBA90DF275648F8C275 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@305da695, comparator=cascading.tuple.hadoop.util.TupleComparator@56d23bef, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:13,508 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B8EA621789834CBA90DF275648F8C275: , UsingHashComparator=false
2016-05-18 00:13:13,644 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:13,644 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:13,645 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B8EA621789834CBA90DF275648F8C275: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:13,647 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B8EA621789834CBA90DF275648F8C275: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:13,653 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:13,657 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CA8D8362F2F942B29BF2CD6A11E8A1AF, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:13:13,668 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,668 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA8D8362F2F942B29BF2CD6A11E8A1AF, mem on close (mb), free: 62, total: 281, max: 672
2016-05-18 00:13:13,669 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B8EA621789834CBA90DF275648F8C275: Starting flush of map output
2016-05-18 00:13:13,669 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B8EA621789834CBA90DF275648F8C275: Span0.length = 35, perItem = 47
2016-05-18 00:13:13,672 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B8EA621789834CBA90DF275648F8C275: done sorting span=0, length=35, time=2
2016-05-18 00:13:13,675 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B8EA621789834CBA90DF275648F8C275: Heap = SpanIterator<0:34> (span=Span[16777216,1676]),
2016-05-18 00:13:13,681 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B8EA621789834CBA90DF275648F8C275: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_00_000000_0_10003_0/file.out
2016-05-18 00:13:13,716 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:13 Completed running task attempt: attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,719 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:13:13,721 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530391803_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:13:13,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530391803_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463530393658, Process_Duration=10, Process_End_Time=1463530393668, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-18 00:13:13,732 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530391803_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:13,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, taskAttemptId=attempt_1463530391803_0001_1_00_000000_0, creationTime=1463530393130, allocationTime=1463530393226, startTime=1463530393242, finishTime=1463530393731, timeTaken=489, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:13,733 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:13,733 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:13,734 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:13,734 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000001
2016-05-18 00:13:13,748 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530391803_0001_1_00_000000_0
2016-05-18 00:13:13,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_FINISHED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, taskId=task_1463530391803_0001_1_00_000000, startTime=1463530393242, finishTime=1463530393750, timeTaken=508, status=SUCCEEDED, successfulAttemptID=attempt_1463530391803_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530393658, Process_Duration=10, Process_End_Time=1463530393668, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7
2016-05-18 00:13:13,773 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:13,773 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:13,782 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_FINISHED]: vertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, vertexId=vertex_1463530391803_0001_1_00, initRequestedTime=1463530392816, initedTime=1463530392996, startRequestedTime=1463530392852, startedTime=1463530393090, finishTime=1463530393774, timeTaken=684, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530393658, Process_Duration=10, Process_End_Time=1463530393668, Read_Duration=1, Tuples_Read=7, Tuples_Written=35, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463530393242, firstTasksToStart=[ task_1463530391803_0001_1_00_000000 ], lastTaskFinishTime=1463530393731, lastTasksToFinish=[ task_1463530391803_0001_1_00_000000 ], minTaskDuration=489, maxTaskDuration=489, avgTaskDuration=489.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530391803_0001_1_00_000000 ], longestDurationTasks=[ task_1463530391803_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:13,783 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:13,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530391803_0001_00_000001, stoppedTime=1463530393785, exitStatus=0
2016-05-18 00:13:13,786 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] attempt: attempt_1463530391803_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:13,786 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B8EA621789834CBA90DF275648F8C275
2016-05-18 00:13:13,786 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:13,787 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:13,787 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, destinationVertexName=B8EA621789834CBA90DF275648F8C275. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:13,788 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530391803_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:13:13,788 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:13,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_STARTED]: vertexName=B8EA621789834CBA90DF275648F8C275, taskId=task_1463530391803_0001_1_01_000000, scheduledTime=1463530393789, launchTime=1463530393789
2016-05-18 00:13:13,790 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:13,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530391803_0001_00_000002, launchTime=1463530393792
2016-05-18 00:13:13,824 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530391803_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:13:13,825 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000001: 87 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:13,825 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530391803_0001_00_000001, Exiting
2016-05-18 00:13:13,825 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530391803_0001_00_000001
2016-05-18 00:13:13,826 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530391803_0001_00_000001
2016-05-18 00:13:13,826 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000002
2016-05-18 00:13:13,830 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:13,831 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530391803_0001_1_01_000000_0] started. Is using containerId: [container_1463530391803_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:13:13,832 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B8EA621789834CBA90DF275648F8C275, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0, startTime=1463530393830, containerId=container_1463530391803_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:13:13 Starting to run new task attempt: attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:13,832 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:13,832 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:13,832 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:13,835 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, VertexName: B8EA621789834CBA90DF275648F8C275, VertexParallelism: 1, TaskAttemptID:attempt_1463530391803_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CA8D8362F2F942B29BF2CD6A11E8A1AF, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EFFA0841F266450AB4F0E09C4C3D6002, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:13,835 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:13,833 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530391803_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:13,836 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:13,838 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530391803_0001_00_000001 completed successfully
2016-05-18 00:13:13,841 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530391803_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:13,844 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530391803_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:13,860 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:13,869 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:13,885 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B8EA621789834CBA90DF275648F8C275, ordinal: 1
2016-05-18 00:13:13,886 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8EA621789834CBA90DF275648F8C275, mem on start (mb), free: 153, total: 281, max: 672
2016-05-18 00:13:13,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:13,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:13,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:13,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:13,888 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EFFA0841F266450AB4F0E09C4C3D6002:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [CA8D8362F2F942B29BF2CD6A11E8A1AF:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:13,888 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:13,889 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:13,890 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:13,890 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EFFA0841F266450AB4F0E09C4C3D6002:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [CA8D8362F2F942B29BF2CD6A11E8A1AF:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:13,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:13,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CA8D8362F2F942B29BF2CD6A11E8A1AF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:13,893 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:13,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:13,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:13,897 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:13,916 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CA8D8362F2F942B29BF2CD6A11E8A1AF with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:13,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CA8D8362F2F942B29BF2CD6A11E8A1AF
2016-05-18 00:13:13,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:13,922 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:13,925 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]]
2016-05-18 00:13:13,930 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:13,933 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 8B241D0ED4E54E05813803A2240216F1
2016-05-18 00:13:13,934 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: 2D302C1C1F92435B9EB66DBB7EDB2953
2016-05-18 00:13:13,934 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2D302C1C1F92435B9EB66DBB7EDB2953
2016-05-18 00:13:13,938 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530391803_0001_1_00_000000_0_10003} to DISK_DIRECT, csize=1089, dsize=1085, EndTime=1463530393938, TimeTaken=2, Rate=0.52 MB/s
2016-05-18 00:13:13,938 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CA8D8362F2F942B29BF2CD6A11E8A1AF
2016-05-18 00:13:13,939 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:13,940 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CA8D8362F2F942B29BF2CD6A11E8A1AF, wasInterrupted=false
2016-05-18 00:13:13,943 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: Already shutdown. Ignoring fetch complete
2016-05-18 00:13:13,943 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:13,944 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EFFA0841F266450AB4F0E09C4C3D6002 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@676827d6, comparator=cascading.tuple.hadoop.util.TupleComparator@5605afe5, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:13,944 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:13,945 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EFFA0841F266450AB4F0E09C4C3D6002: , UsingHashComparator=false
2016-05-18 00:13:13,945 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_00_000000_0_10003/file.out, len=1089, isLocal=true
2016-05-18 00:13:13,947 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 1089 bytes from disk
2016-05-18 00:13:13,948 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:13,952 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:13,955 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-18 00:13:13,956 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CA8D8362F2F942B29BF2CD6A11E8A1AF
2016-05-18 00:13:13,965 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: Shuffle Runner thread complete
2016-05-18 00:13:13,965 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:13,966 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:13,966 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EFFA0841F266450AB4F0E09C4C3D6002: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:13,966 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EFFA0841F266450AB4F0E09C4C3D6002: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:13,967 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'l'][numSelfJoins:1]] 8B241D0ED4E54E05813803A2240216F1, for 1 inputs
2016-05-18 00:13:13,974 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:13,974 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:13,978 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B8EA621789834CBA90DF275648F8C275, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:13,979 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:14,023 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:14,023 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8EA621789834CBA90DF275648F8C275, mem on close (mb), free: 46, total: 281, max: 672
2016-05-18 00:13:14,023 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CA8D8362F2F942B29BF2CD6A11E8A1AF
2016-05-18 00:13:14,024 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CA8D8362F2F942B29BF2CD6A11E8A1AF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:14,024 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EFFA0841F266450AB4F0E09C4C3D6002: Starting flush of map output
2016-05-18 00:13:14,025 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EFFA0841F266450AB4F0E09C4C3D6002: Span0.length = 174, perItem = 75
2016-05-18 00:13:14,052 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EFFA0841F266450AB4F0E09C4C3D6002: done sorting span=0, length=174, time=27
2016-05-18 00:13:14,052 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EFFA0841F266450AB4F0E09C4C3D6002: Heap = SpanIterator<0:173> (span=Span[16777216,13104]),
2016-05-18 00:13:14,053 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EFFA0841F266450AB4F0E09C4C3D6002: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_01_000000_0_10006_0/file.out
2016-05-18 00:13:14,060 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:14 Completed running task attempt: attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:14,061 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:13:14,062 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530391803_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:13:14,063 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530391803_0001_1_01_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=47, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530393979, Process_Duration=44, Process_End_Time=1463530394023, Read_Duration=6, Tuples_Read=35, Tuples_Written=174, Write_Duration=13]]
2016-05-18 00:13:14,065 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530391803_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,065 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,065 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B8EA621789834CBA90DF275648F8C275, taskAttemptId=attempt_1463530391803_0001_1_01_000000_0, creationTime=1463530393790, allocationTime=1463530393791, startTime=1463530393830, finishTime=1463530394065, timeTaken=235, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:14,065 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:14,066 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,066 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530391803_0001_1_01_000000_0
2016-05-18 00:13:14,066 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000002
2016-05-18 00:13:14,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_FINISHED]: vertexName=B8EA621789834CBA90DF275648F8C275, taskId=task_1463530391803_0001_1_01_000000, startTime=1463530393830, finishTime=1463530394067, timeTaken=237, status=SUCCEEDED, successfulAttemptID=attempt_1463530391803_0001_1_01_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=47, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530393979, Process_Duration=44, Process_End_Time=1463530394023, Read_Duration=6, Tuples_Read=35, Tuples_Written=174, Write_Duration=13
2016-05-18 00:13:14,070 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:14,086 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:14,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_FINISHED]: vertexName=B8EA621789834CBA90DF275648F8C275, vertexId=vertex_1463530391803_0001_1_01, initRequestedTime=1463530392852, initedTime=1463530392933, startRequestedTime=1463530393100, startedTime=1463530393138, finishTime=1463530394087, timeTaken=949, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=47, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530393979, Process_Duration=44, Process_End_Time=1463530394023, Read_Duration=6, Tuples_Read=35, Tuples_Written=174, Write_Duration=13, vertexStats=firstTaskStartTime=1463530393830, firstTasksToStart=[ task_1463530391803_0001_1_01_000000 ], lastTaskFinishTime=1463530394065, lastTasksToFinish=[ task_1463530391803_0001_1_01_000000 ], minTaskDuration=235, maxTaskDuration=235, avgTaskDuration=235.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530391803_0001_1_01_000000 ], longestDurationTasks=[ task_1463530391803_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:14,094 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:14,095 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530391803_0001_00_000002, stoppedTime=1463530394095, exitStatus=0
2016-05-18 00:13:14,096 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] attempt: attempt_1463530391803_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:14,101 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:14,102 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:14,101 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EFFA0841F266450AB4F0E09C4C3D6002
2016-05-18 00:13:14,102 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530391803_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:13:14,102 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B8EA621789834CBA90DF275648F8C275, destinationVertexName=EFFA0841F266450AB4F0E09C4C3D6002. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:14,103 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:14,105 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_STARTED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, taskId=task_1463530391803_0001_1_02_000000, scheduledTime=1463530394105, launchTime=1463530394105
2016-05-18 00:13:14,106 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:14,106 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530391803_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:13:14,107 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000002: 40 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:14,107 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530391803_0001_00_000002, Exiting
2016-05-18 00:13:14,107 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530391803_0001_00_000002
2016-05-18 00:13:14,107 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530391803_0001_00_000002
2016-05-18 00:13:14,108 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530391803_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:14,108 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530391803_0001_00_000002 completed successfully
2016-05-18 00:13:14,111 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530391803_0001_00_000003, launchTime=1463530394110
2016-05-18 00:13:14,112 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000003
2016-05-18 00:13:14,113 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530391803_0001_1_02_000000_0] started. Is using containerId: [container_1463530391803_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 00:13:14,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0, startTime=1463530394112, containerId=container_1463530391803_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 00:13:14,114 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14 Starting to run new task attempt: attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,114 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,115 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,115 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:14,115 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EFFA0841F266450AB4F0E09C4C3D6002, VertexParallelism: 1, TaskAttemptID:attempt_1463530391803_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B8EA621789834CBA90DF275648F8C275, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EDFB22285A7E4CB5A5C46FB0F9083336, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:14,116 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:14,116 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,123 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530391803_0001_1_02_000000_0 1 events [0,1) total 1 vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:14,124 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530391803_0001_1_02_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:14,126 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:14,126 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:14,140 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EFFA0841F266450AB4F0E09C4C3D6002, ordinal: 2
2016-05-18 00:13:14,140 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EFFA0841F266450AB4F0E09C4C3D6002, mem on start (mb), free: 259, total: 391, max: 672
2016-05-18 00:13:14,141 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:14,141 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:14,141 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:14,142 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:14,143 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B8EA621789834CBA90DF275648F8C275:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [EDFB22285A7E4CB5A5C46FB0F9083336:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:14,143 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:14,144 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:14,144 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:14,145 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B8EA621789834CBA90DF275648F8C275:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [EDFB22285A7E4CB5A5C46FB0F9083336:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:14,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:14,147 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B8EA621789834CBA90DF275648F8C275: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:14,147 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B8EA621789834CBA90DF275648F8C275: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:14,149 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B8EA621789834CBA90DF275648F8C275 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:14,149 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:14,149 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:14,149 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B8EA621789834CBA90DF275648F8C275 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:14,152 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B8EA621789834CBA90DF275648F8C275
2016-05-18 00:13:14,152 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:14,154 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,154 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8EA621789834CBA90DF275648F8C275: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:14,159 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530391803_0001_1_01_000000_0_10006} to DISK_DIRECT, csize=7519, dsize=7515, EndTime=1463530394159, TimeTaken=2, Rate=3.59 MB/s
2016-05-18 00:13:14,160 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B8EA621789834CBA90DF275648F8C275
2016-05-18 00:13:14,160 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:14,160 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B8EA621789834CBA90DF275648F8C275, wasInterrupted=false
2016-05-18 00:13:14,160 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-18 00:13:14,163 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - B8EA621789834CBA90DF275648F8C275: Already shutdown. Ignoring fetch complete
2016-05-18 00:13:14,163 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:14,163 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:14,164 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_01_000000_0_10006/file.out, len=7519, isLocal=true
2016-05-18 00:13:14,164 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 7519 bytes from disk
2016-05-18 00:13:14,165 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:14,165 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:14,166 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: 2D302C1C1F92435B9EB66DBB7EDB2953
2016-05-18 00:13:14,167 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'n1', 'n2']], id: FCB609D53A514E889A6BE0D2DD1821AD
2016-05-18 00:13:14,167 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-18 00:13:14,167 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B8EA621789834CBA90DF275648F8C275
2016-05-18 00:13:14,167 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] FCB609D53A514E889A6BE0D2DD1821AD
2016-05-18 00:13:14,167 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B8EA621789834CBA90DF275648F8C275: Shuffle Runner thread complete
2016-05-18 00:13:14,172 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EDFB22285A7E4CB5A5C46FB0F9083336 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3bbc58a7, comparator=cascading.tuple.hadoop.util.TupleComparator@4a605a94, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:14,172 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EDFB22285A7E4CB5A5C46FB0F9083336: , UsingHashComparator=false
2016-05-18 00:13:14,187 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:14,187 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:14,187 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EDFB22285A7E4CB5A5C46FB0F9083336: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:14,187 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EDFB22285A7E4CB5A5C46FB0F9083336: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:14,188 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] 2D302C1C1F92435B9EB66DBB7EDB2953, for 1 inputs
2016-05-18 00:13:14,189 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EFFA0841F266450AB4F0E09C4C3D6002, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:14,190 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B8EA621789834CBA90DF275648F8C275: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:14,216 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,216 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EFFA0841F266450AB4F0E09C4C3D6002, mem on close (mb), free: 155, total: 391, max: 672
2016-05-18 00:13:14,216 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B8EA621789834CBA90DF275648F8C275
2016-05-18 00:13:14,216 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8EA621789834CBA90DF275648F8C275: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:14,216 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EDFB22285A7E4CB5A5C46FB0F9083336: Starting flush of map output
2016-05-18 00:13:14,217 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EDFB22285A7E4CB5A5C46FB0F9083336: Span0.length = 87, perItem = 72
2016-05-18 00:13:14,225 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EDFB22285A7E4CB5A5C46FB0F9083336: done sorting span=0, length=87, time=8
2016-05-18 00:13:14,225 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EDFB22285A7E4CB5A5C46FB0F9083336: Heap = SpanIterator<0:86> (span=Span[16777216,6291]),
2016-05-18 00:13:14,226 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EDFB22285A7E4CB5A5C46FB0F9083336: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_02_000000_0_10009_0/file.out
2016-05-18 00:13:14,231 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:14 Completed running task attempt: attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,232 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0, askedToStop=false
2016-05-18 00:13:14,233 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530391803_0001_1_02_000000_0, stopRequested=false
2016-05-18 00:13:14,233 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530391803_0001_1_02_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530394189, Process_Duration=27, Process_End_Time=1463530394216, Read_Duration=8, Tuples_Read=174, Tuples_Written=87, Write_Duration=4]]
2016-05-18 00:13:14,235 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, taskAttemptId=attempt_1463530391803_0001_1_02_000000_0, creationTime=1463530394105, allocationTime=1463530394108, startTime=1463530394112, finishTime=1463530394235, timeTaken=123, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:14,235 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530391803_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,236 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,237 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:14,237 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,236 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530391803_0001_1_02_000000_0
2016-05-18 00:13:14,237 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000003
2016-05-18 00:13:14,240 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_FINISHED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, taskId=task_1463530391803_0001_1_02_000000, startTime=1463530394112, finishTime=1463530394237, timeTaken=125, status=SUCCEEDED, successfulAttemptID=attempt_1463530391803_0001_1_02_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530394189, Process_Duration=27, Process_End_Time=1463530394216, Read_Duration=8, Tuples_Read=174, Tuples_Written=87, Write_Duration=4
2016-05-18 00:13:14,240 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:14,241 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:14,245 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_FINISHED]: vertexName=EFFA0841F266450AB4F0E09C4C3D6002, vertexId=vertex_1463530391803_0001_1_02, initRequestedTime=1463530392945, initedTime=1463530392994, startRequestedTime=1463530393147, startedTime=1463530393159, finishTime=1463530394241, timeTaken=1082, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530394189, Process_Duration=27, Process_End_Time=1463530394216, Read_Duration=8, Tuples_Read=174, Tuples_Written=87, Write_Duration=4, vertexStats=firstTaskStartTime=1463530394112, firstTasksToStart=[ task_1463530391803_0001_1_02_000000 ], lastTaskFinishTime=1463530394235, lastTasksToFinish=[ task_1463530391803_0001_1_02_000000 ], minTaskDuration=123, maxTaskDuration=123, avgTaskDuration=123.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530391803_0001_1_02_000000 ], longestDurationTasks=[ task_1463530391803_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:14,245 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:14,246 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530391803_0001_00_000003, stoppedTime=1463530394245, exitStatus=0
2016-05-18 00:13:14,246 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] attempt: attempt_1463530391803_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:14,247 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EDFB22285A7E4CB5A5C46FB0F9083336
2016-05-18 00:13:14,247 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EFFA0841F266450AB4F0E09C4C3D6002, destinationVertexName=EDFB22285A7E4CB5A5C46FB0F9083336. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:14,248 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:14,250 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:14,250 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530391803_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 00:13:14,250 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000003: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:14,250 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:14,250 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530391803_0001_00_000003, Exiting
2016-05-18 00:13:14,251 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530391803_0001_00_000003
2016-05-18 00:13:14,251 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530391803_0001_00_000003
2016-05-18 00:13:14,251 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530391803_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:14,252 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530391803_0001_00_000003 completed successfully
2016-05-18 00:13:14,252 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463530391803_0001_00_000003
2016-05-18 00:13:14,253 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_STARTED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, taskId=task_1463530391803_0001_1_03_000000, scheduledTime=1463530394252, launchTime=1463530394252
2016-05-18 00:13:14,253 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:14,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530391803_0001_00_000004, launchTime=1463530394257
2016-05-18 00:13:14,258 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000004
2016-05-18 00:13:14,259 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530391803_0001_1_03_000000_0] started. Is using containerId: [container_1463530391803_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-18 00:13:14,260 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0, startTime=1463530394259, containerId=container_1463530391803_0001_00_000004, nodeId=127.0.0.1:0
2016-05-18 00:13:14,260 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14 Starting to run new task attempt: attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,261 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,262 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,262 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:14,262 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, VertexName: EDFB22285A7E4CB5A5C46FB0F9083336, VertexParallelism: 1, TaskAttemptID:attempt_1463530391803_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EFFA0841F266450AB4F0E09C4C3D6002, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A3F040E13A344444A10F613545D9C067, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:13:14,262 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:14,263 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,264 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530391803_0001_1_03_000000_0 1 events [0,1) total 1 vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:14,265 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530391803_0001_1_03_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:14,269 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:14,273 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:14,284 INFO output.MROutput (MROutput.java:initialize(421)) - A3F040E13A344444A10F613545D9C067: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:13:14,290 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EDFB22285A7E4CB5A5C46FB0F9083336, ordinal: 3
2016-05-18 00:13:14,291 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDFB22285A7E4CB5A5C46FB0F9083336, mem on start (mb), free: 150, total: 391, max: 672
2016-05-18 00:13:14,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:14,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:14,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:14,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:14,291 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A3F040E13A344444A10F613545D9C067:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [EFFA0841F266450AB4F0E09C4C3D6002:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:14,292 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:14,292 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:14,293 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:13:14,293 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A3F040E13A344444A10F613545D9C067:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [EFFA0841F266450AB4F0E09C4C3D6002:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:14,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:14,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EFFA0841F266450AB4F0E09C4C3D6002 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:14,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:14,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:14,294 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EFFA0841F266450AB4F0E09C4C3D6002: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:14,295 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EFFA0841F266450AB4F0E09C4C3D6002: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:14,296 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EFFA0841F266450AB4F0E09C4C3D6002 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:14,298 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EFFA0841F266450AB4F0E09C4C3D6002
2016-05-18 00:13:14,298 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:14,299 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,299 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EFFA0841F266450AB4F0E09C4C3D6002: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:14,303 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'n1', 'n2']]
2016-05-18 00:13:14,305 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] streamed: true, id: FCB609D53A514E889A6BE0D2DD1821AD
2016-05-18 00:13:14,305 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"], id: A3F040E13A344444A10F613545D9C067
2016-05-18 00:13:14,306 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530391803_0001_1_02_000000_0_10009} to DISK_DIRECT, csize=4344, dsize=4340, EndTime=1463530394306, TimeTaken=5, Rate=0.83 MB/s
2016-05-18 00:13:14,306 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite"]
2016-05-18 00:13:14,306 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EFFA0841F266450AB4F0E09C4C3D6002
2016-05-18 00:13:14,306 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:14,310 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EFFA0841F266450AB4F0E09C4C3D6002, wasInterrupted=false
2016-05-18 00:13:14,310 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'n1', 'n2']] FCB609D53A514E889A6BE0D2DD1821AD, for 1 inputs
2016-05-18 00:13:14,310 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:14,312 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - EFFA0841F266450AB4F0E09C4C3D6002: Already shutdown. Ignoring fetch complete
2016-05-18 00:13:14,313 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:14,313 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001_wd/localmode-local-dir/output/attempt_1463530391803_0001_1_02_000000_0_10009/file.out, len=4344, isLocal=true
2016-05-18 00:13:14,314 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 4344 bytes from disk
2016-05-18 00:13:14,314 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:14,314 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:14,314 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-05-18 00:13:14,314 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EFFA0841F266450AB4F0E09C4C3D6002
2016-05-18 00:13:14,315 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EFFA0841F266450AB4F0E09C4C3D6002: Shuffle Runner thread complete
2016-05-18 00:13:14,315 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EDFB22285A7E4CB5A5C46FB0F9083336, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:13:14,316 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EFFA0841F266450AB4F0E09C4C3D6002: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:14,329 INFO output.MROutput (MROutput.java:close(526)) - A3F040E13A344444A10F613545D9C067 closed
2016-05-18 00:13:14,330 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463530391803_0001_1_03_000000_0 given a go for committing the task output.
2016-05-18 00:13:14,331 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,331 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDFB22285A7E4CB5A5C46FB0F9083336, mem on close (mb), free: 147, total: 391, max: 672
2016-05-18 00:13:14,332 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EFFA0841F266450AB4F0E09C4C3D6002
2016-05-18 00:13:14,332 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EFFA0841F266450AB4F0E09C4C3D6002: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:14,332 INFO output.MROutput (MROutput.java:close(526)) - A3F040E13A344444A10F613545D9C067 closed
2016-05-18 00:13:14,332 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:14 Completed running task attempt: attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,333 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0, askedToStop=false
2016-05-18 00:13:14,333 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530391803_0001_1_03_000000_0, stopRequested=false
2016-05-18 00:13:14,334 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530391803_0001_1_03_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530394315, Process_Duration=16, Process_End_Time=1463530394331, Read_Duration=7, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-18 00:13:14,336 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530391803_0001_1_03_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,336 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:14,337 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:14,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, taskAttemptId=attempt_1463530391803_0001_1_03_000000_0, creationTime=1463530394253, allocationTime=1463530394256, startTime=1463530394259, finishTime=1463530394336, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:14,338 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,338 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530391803_0001_00_000004
2016-05-18 00:13:14,338 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530391803_0001_1_03_000000_0
2016-05-18 00:13:14,340 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:TASK_FINISHED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, taskId=task_1463530391803_0001_1_03_000000, startTime=1463530394259, finishTime=1463530394338, timeTaken=79, status=SUCCEEDED, successfulAttemptID=attempt_1463530391803_0001_1_03_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530394315, Process_Duration=16, Process_End_Time=1463530394331, Read_Duration=7, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-18 00:13:14,342 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:14,342 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336]
2016-05-18 00:13:14,345 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:VERTEX_FINISHED]: vertexName=EDFB22285A7E4CB5A5C46FB0F9083336, vertexId=vertex_1463530391803_0001_1_03, initRequestedTime=1463530392999, initedTime=1463530393037, startRequestedTime=1463530393162, startedTime=1463530393163, finishTime=1463530394342, timeTaken=1179, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=16, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530394315, Process_Duration=16, Process_End_Time=1463530394331, Read_Duration=7, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463530394259, firstTasksToStart=[ task_1463530391803_0001_1_03_000000 ], lastTaskFinishTime=1463530394336, lastTasksToFinish=[ task_1463530391803_0001_1_03_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530391803_0001_1_03_000000 ], longestDurationTasks=[ task_1463530391803_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:14,345 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:14,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530391803_0001_00_000004, stoppedTime=1463530394346, exitStatus=0
2016-05-18 00:13:14,346 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:14,346 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:14,346 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530391803_0001_1_00 [CA8D8362F2F942B29BF2CD6A11E8A1AF]
2016-05-18 00:13:14,346 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530391803_0001_1_01 [B8EA621789834CBA90DF275648F8C275]
2016-05-18 00:13:14,346 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530391803_0001_1_02 [EFFA0841F266450AB4F0E09C4C3D6002]
2016-05-18 00:13:14,348 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463530391803_0001_1
2016-05-18 00:13:14,349 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530391803_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-18 00:13:14,350 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530391803_0001_00_000004: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:14,350 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530391803_0001_00_000004, Exiting
2016-05-18 00:13:14,350 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530391803_0001_00_000004
2016-05-18 00:13:14,350 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530391803_0001_00_000004
2016-05-18 00:13:14,350 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530391803_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:14,350 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530391803_0001_00_000004 completed successfully
2016-05-18 00:13:14,354 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463530391803_0001_1
2016-05-18 00:13:14,354 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: A3F040E13A344444A10F613545D9C067 for vertex: vertex_1463530391803_0001_1_03 [EDFB22285A7E4CB5A5C46FB0F9083336], outputName: A3F040E13A344444A10F613545D9C067
2016-05-18 00:13:14,357 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530391803_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:13:14,358 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463530391803_0001_00_000004
2016-05-18 00:13:14,365 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:A3F040E13A344444A10F613545D9C067 of vertex/vertexGroup:EDFB22285A7E4CB5A5C46FB0F9083336 isVertexGroupOutput:false
2016-05-18 00:13:14,365 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:14,376 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463530391803_0001_1, queueSize=0
2016-05-18 00:13:14,378 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530391803_0001_1][Event:DAG_FINISHED]: dagId=dag_1463530391803_0001_1, startTime=1463530392814, finishTime=1463530394365, timeTaken=1551, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=3760, AM_GC_TIME_MILLIS=47, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=114, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=3, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=78, MERGE_PHASE_TIME=103, FIRST_EVENT_RECEIVED=49, LAST_EVENT_RECEIVED=49, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854121576141, Process_Duration=97, Process_End_Time=5854121576238, Read_Duration=22, Tuples_Read=303, Tuples_Written=317, Write_Duration=20, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-18 00:13:14,379 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463530391803_0001_1 finished with state: SUCCEEDED
2016-05-18 00:13:14,379 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530391803_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:13:14 Completed Dag: dag_1463530391803_0001_1
2016-05-18 00:13:14,382 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463530391803_0001_1, dagState=SUCCEEDED
2016-05-18 00:13:14,382 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530391803_0001_1_post
2016-05-18 00:13:14,383 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:14,383 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:13:14,383 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:13:14,384 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, with id=dag_1463530391803_0001_1
2016-05-18 00:13:14,387 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56] euclidean/(1/1) ...osite/euclidean/composite, with id=dag_1463530391803_0001_1
2016-05-18 00:13:14,391 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...osite/euclidean/composite, applicationId=application_1463530391803_0001
2016-05-18 00:13:14,396 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:13:14,396 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:13:14,396 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:13:14,396 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:13:14,396 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:13:14,397 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:13:14,400 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:13:14,400 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:13:14,400 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:13:14,400 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:13:14,400 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:13:14,401 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:13:14,401 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:13:14,403 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:13:14,407 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/3ED271D6312D460AB32BE18FB5E68600/AAB4576689E943539839714939118E56/.tez/application_1463530391803_0001
2016-05-18 00:13:14,409 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite/_temporary
2016-05-18 00:13:14,410 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:02.810
2016-05-18 00:13:14,530 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceComposite/euclidean/composite
2016-05-18 00:13:14,590 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:13:14,591 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:13:14,696 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.055
2016-05-18 00:13:14,742 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.100
2016-05-18 00:13:14,743 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:14,743 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:14,744 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:13:14,777 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt
2016-05-18 00:13:14,811 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-05-18 00:13:15,064 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] starting
2016-05-18 00:13:15,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:15,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-05-18 00:13:15,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] parallel execution of steps is enabled: false
2016-05-18 00:13:15,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] executing total steps: 1
2016-05-18 00:13:15,066 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] allocating management threads: 1
2016-05-18 00:13:15,066 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] starting step: (1/1) ...mposite/pearson/composite
2016-05-18 00:13:15,067 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [pearson] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:13:15,084 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] tez session mode enabled: true
2016-05-18 00:13:15,094 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:15,095 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:13:15,095 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:13:15,105 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001 doesn't exist and is created
2016-05-18 00:13:15,125 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:13:15,125 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:15,126 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd
2016-05-18 00:13:15,137 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463530395095_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:15,138 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:15,138 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:15,139 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:13:15,139 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:13:15,139 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:13:15,140 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:13:15,141 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:13:15,144 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:13:15,144 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:13:15,145 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:13:15,146 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:15,148 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:15,148 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:13:15,149 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:13:15,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:13:15,156 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:13:15,157 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:13:15,157 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-log-dir
2016-05-18 00:13:15,158 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-log-dir/history.txt.appattempt_1463530395095_0001_000000, maxErrors=10
2016-05-18 00:13:15,158 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:13:15,159 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463530395095_0001_000000, appSubmitTime=1463530395137, launchTime=1463530395137
2016-05-18 00:13:15,167 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:13:15,176 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47327
2016-05-18 00:13:15,177 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463530395095
2016-05-18 00:13:15,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463530395095_0001_000000, startTime=1463530395180
2016-05-18 00:13:15,181 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:13:15,226 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:13:15,226 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:13:15,227 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463530395095_0001, dagName=[63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite
2016-05-18 00:13:15,261 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite
2016-05-18 00:13:15,262 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:13:15,263 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463530395095_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-log-dir/dag_1463530395095_0001_1.dot
2016-05-18 00:13:15,266 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-log-dir/dag_1463530395095_0001_1-tez-dag.pb.txt
2016-05-18 00:13:15,283 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530395095_0001_1
2016-05-18 00:13:15,283 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,283 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite
2016-05-18 00:13:15 Running Dag: dag_1463530395095_0001_1
2016-05-18 00:13:15,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463530395095_0001_1, submitTime=1463530395261
2016-05-18 00:13:15,296 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001/tez-conf.pb" } size: 36897 timestamp: 1463530395000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:15,296 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,296 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:15,297 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001/tez-conf.pb" } size: 36897 timestamp: 1463530395000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:15,299 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:15,300 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001/tez-conf.pb" } size: 36897 timestamp: 1463530395000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:15,300 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:15,302 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001/tez-conf.pb" } size: 36897 timestamp: 1463530395000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:15,302 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,303 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:15,304 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:13:15,304 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463530395095_0001_1, initTime=1463530395304
2016-05-18 00:13:15,305 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530395095_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:13:15,305 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:13:15,306 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:DAG_STARTED]: dagID=dag_1463530395095_0001_1, startTime=1463530395306
2016-05-18 00:13:15,306 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530395095_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:13:15,306 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D43FEC2C533440559297E3A2FCC21D83 : {63C033FB790D458FAFED23BC92853F4A={InputName=63C033FB790D458FAFED23BC92853F4A}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:13:15,306 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 63C033FB790D458FAFED23BC92853F4A, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:13:15,306 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,307 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,307 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,307 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,307 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:13:15,308 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,308 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463530395095_0001, dagName=[63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite
2016-05-18 00:13:15,315 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:15,315 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,315 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 63C033FB790D458FAFED23BC92853F4A on vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,316 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,319 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530395095_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:15,319 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, vertexId=vertex_1463530395095_0001_1_01, initRequestedTime=1463530395308, initedTime=1463530395319, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:15,319 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:15,320 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,322 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 63C033FB790D458FAFED23BC92853F4A asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:13:15,329 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:15,329 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,329 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:13:15,330 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 828 Min splitLength: 52428800 New desired splits: 1 Total length: 828 Original splits: 2
2016-05-18 00:13:15,330 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 828 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 828 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:13:15,330 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,331 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530395095_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:15,332 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, vertexId=vertex_1463530395095_0001_1_02, initRequestedTime=1463530395320, initedTime=1463530395332, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:15,333 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:15,333 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,331 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:13:15,335 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 63C033FB790D458FAFED23BC92853F4A on vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,340 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:15,340 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,341 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,341 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] submitted tez dag to app master: application_1463530395095_0001, with dag id: dag_1463530395095_0001_1
2016-05-18 00:13:15,342 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530395095_0001_1_03, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:15,342 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6], numAdditionalOutputs=1
2016-05-18 00:13:15,342 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=5990AD4214364C3C8BDD323947B28F35, vertex=vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:13:15,345 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A7D1737F6E25420CB295B88F11AFAFE6:5990AD4214364C3C8BDD323947B28F35 using oldmapred API
2016-05-18 00:13:15,345 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=5990AD4214364C3C8BDD323947B28F35, vertexName=A7D1737F6E25420CB295B88F11AFAFE6, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:13:15,347 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, vertexId=vertex_1463530395095_0001_1_03, initRequestedTime=1463530395333, initedTime=1463530395342, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:15,347 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:15,358 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:15,359 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {63C033FB790D458FAFED23BC92853F4A=forAllWorkUnits=true, update=[1]}
2016-05-18 00:13:15,360 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] parallelism set to 1
2016-05-18 00:13:15,360 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:15,362 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, vertexId=vertex_1463530395095_0001_1_00, initRequestedTime=1463530395306, initedTime=1463530395362, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:13:15,362 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] with distanceFromRoot: 0
2016-05-18 00:13:15,363 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:13:15,363 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530395095_0001_1_00, startRequestedTime=1463530395307, startedTime=1463530395363
2016-05-18 00:13:15,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530395095_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:13:15,364 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:15,364 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D43FEC2C533440559297E3A2FCC21D83
2016-05-18 00:13:15,364 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,365 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530395095_0001_1_00 for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] numStartedSources: 1 numSources: 1
2016-05-18 00:13:15,365 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] with distanceFromRoot: 1
2016-05-18 00:13:15,366 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530395095_0001_1_01, startRequestedTime=1463530395365, startedTime=1463530395365
2016-05-18 00:13:15,367 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:15,368 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530395095_0001_1_01 for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] numStartedSources: 1 numSources: 1
2016-05-18 00:13:15,368 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] with distanceFromRoot: 2
2016-05-18 00:13:15,369 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D27F6E9D4C064764BAA82EE0A9034C00 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:15,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_STARTED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, taskId=task_1463530395095_0001_1_00_000000, scheduledTime=1463530395369, launchTime=1463530395369
2016-05-18 00:13:15,371 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D43FEC2C533440559297E3A2FCC21D83 in vertex: D27F6E9D4C064764BAA82EE0A9034C00 numBipartiteSourceTasks: 1
2016-05-18 00:13:15,373 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530395095_0001_1_02, startRequestedTime=1463530395368, startedTime=1463530395372
2016-05-18 00:13:15,374 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:15,374 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CC7C82307BB441C4A2C97CA7B9F46328 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:15,374 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,375 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D27F6E9D4C064764BAA82EE0A9034C00 in vertex: CC7C82307BB441C4A2C97CA7B9F46328 numBipartiteSourceTasks: 1
2016-05-18 00:13:15,375 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530395095_0001_1_02 for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] numStartedSources: 1 numSources: 1
2016-05-18 00:13:15,376 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] with distanceFromRoot: 3
2016-05-18 00:13:15,377 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530395095_0001_1_03, startRequestedTime=1463530395375, startedTime=1463530395376
2016-05-18 00:13:15,377 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:15,378 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A7D1737F6E25420CB295B88F11AFAFE6 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:15,378 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:13:15,379 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CC7C82307BB441C4A2C97CA7B9F46328 in vertex: A7D1737F6E25420CB295B88F11AFAFE6 numBipartiteSourceTasks: 1
2016-05-18 00:13:15,381 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530395095_0001_00_000001, launchTime=1463530395381
2016-05-18 00:13:15,382 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000001
2016-05-18 00:13:15,382 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530395095_0001_1_00_000000_0] started. Is using containerId: [container_1463530395095_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:13:15,383 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0, startTime=1463530395382, containerId=container_1463530395095_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:13:15,383 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15 Starting to run new task attempt: attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,384 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,385 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,385 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:15,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, VertexName: D43FEC2C533440559297E3A2FCC21D83, VertexParallelism: 1, TaskAttemptID:attempt_1463530395095_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=63C033FB790D458FAFED23BC92853F4A, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D27F6E9D4C064764BAA82EE0A9034C00, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:15,386 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:15,388 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,390 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463530395095_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:13:15,391 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530395095_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:13:15,393 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:15,396 INFO input.MRInput (MRInput.java:initialize(440)) - 63C033FB790D458FAFED23BC92853F4A using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:13:15,396 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 63C033FB790D458FAFED23BC92853F4A
2016-05-18 00:13:15,407 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D43FEC2C533440559297E3A2FCC21D83, ordinal: 0
2016-05-18 00:13:15,407 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D43FEC2C533440559297E3A2FCC21D83, mem on start (mb), free: 196, total: 390, max: 672
2016-05-18 00:13:15,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:15,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:15,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:15,410 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:15,410 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[63C033FB790D458FAFED23BC92853F4A:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D27F6E9D4C064764BAA82EE0A9034C00:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:15,411 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:15,412 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:15,412 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:13:15,412 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[63C033FB790D458FAFED23BC92853F4A:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D27F6E9D4C064764BAA82EE0A9034C00:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:15,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:15,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 63C033FB790D458FAFED23BC92853F4A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:15,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:15,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:15,415 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 63C033FB790D458FAFED23BC92853F4A
2016-05-18 00:13:15,416 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:15,416 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,418 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:0+414, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:414+414], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=828}
2016-05-18 00:13:15,418 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 63C033FB790D458FAFED23BC92853F4A initialized RecordReader from event
2016-05-18 00:13:15,422 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:15,424 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"] streamed: true, id: 63C033FB790D458FAFED23BC92853F4A
2016-05-18 00:13:15,424 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]], id: 901A9D51FB954F1FBFA546C978B3D5E7
2016-05-18 00:13:15,425 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 901A9D51FB954F1FBFA546C978B3D5E7
2016-05-18 00:13:15,429 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D27F6E9D4C064764BAA82EE0A9034C00 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6121658c, comparator=cascading.tuple.hadoop.util.TupleComparator@5791fc25, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:15,429 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D27F6E9D4C064764BAA82EE0A9034C00: , UsingHashComparator=false
2016-05-18 00:13:15,442 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:15,442 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:15,442 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D27F6E9D4C064764BAA82EE0A9034C00: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:15,443 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D27F6E9D4C064764BAA82EE0A9034C00: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:15,443 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:15,444 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D43FEC2C533440559297E3A2FCC21D83, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:15,449 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,450 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D43FEC2C533440559297E3A2FCC21D83, mem on close (mb), free: 94, total: 390, max: 672
2016-05-18 00:13:15,450 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D27F6E9D4C064764BAA82EE0A9034C00: Starting flush of map output
2016-05-18 00:13:15,450 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D27F6E9D4C064764BAA82EE0A9034C00: Span0.length = 35, perItem = 47
2016-05-18 00:13:15,451 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D27F6E9D4C064764BAA82EE0A9034C00: done sorting span=0, length=35, time=1
2016-05-18 00:13:15,452 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D27F6E9D4C064764BAA82EE0A9034C00: Heap = SpanIterator<0:34> (span=Span[16777216,1676]),
2016-05-18 00:13:15,453 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D27F6E9D4C064764BAA82EE0A9034C00: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_00_000000_0_10015_0/file.out
2016-05-18 00:13:15,459 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:15 Completed running task attempt: attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,460 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:13:15,460 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530395095_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:13:15,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530395095_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463530395444, Process_Duration=5, Process_End_Time=1463530395449, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-18 00:13:15,463 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530395095_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,463 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,464 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, taskAttemptId=attempt_1463530395095_0001_1_00_000000_0, creationTime=1463530395372, allocationTime=1463530395380, startTime=1463530395382, finishTime=1463530395463, timeTaken=81, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:15,464 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:15,464 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,464 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530395095_0001_1_00_000000_0
2016-05-18 00:13:15,465 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_FINISHED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, taskId=task_1463530395095_0001_1_00_000000, startTime=1463530395382, finishTime=1463530395464, timeTaken=82, status=SUCCEEDED, successfulAttemptID=attempt_1463530395095_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530395444, Process_Duration=5, Process_End_Time=1463530395449, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7
2016-05-18 00:13:15,466 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:15,466 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,466 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000001
2016-05-18 00:13:15,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_FINISHED]: vertexName=D43FEC2C533440559297E3A2FCC21D83, vertexId=vertex_1463530395095_0001_1_00, initRequestedTime=1463530395306, initedTime=1463530395362, startRequestedTime=1463530395307, startedTime=1463530395363, finishTime=1463530395466, timeTaken=103, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530395444, Process_Duration=5, Process_End_Time=1463530395449, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463530395382, firstTasksToStart=[ task_1463530395095_0001_1_00_000000 ], lastTaskFinishTime=1463530395463, lastTasksToFinish=[ task_1463530395095_0001_1_00_000000 ], minTaskDuration=81, maxTaskDuration=81, avgTaskDuration=81.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530395095_0001_1_00_000000 ], longestDurationTasks=[ task_1463530395095_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:15,468 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:15,468 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530395095_0001_00_000001, stoppedTime=1463530395468, exitStatus=0
2016-05-18 00:13:15,468 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] attempt: attempt_1463530395095_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:15,468 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:15,468 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:15,469 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D27F6E9D4C064764BAA82EE0A9034C00
2016-05-18 00:13:15,469 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D43FEC2C533440559297E3A2FCC21D83, destinationVertexName=D27F6E9D4C064764BAA82EE0A9034C00. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:15,469 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,469 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530395095_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:13:15,470 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_STARTED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, taskId=task_1463530395095_0001_1_01_000000, scheduledTime=1463530395469, launchTime=1463530395469
2016-05-18 00:13:15,470 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,473 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530395095_0001_00_000002, launchTime=1463530395472
2016-05-18 00:13:15,473 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000002
2016-05-18 00:13:15,474 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,474 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530395095_0001_1_01_000000_0] started. Is using containerId: [container_1463530395095_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:13:15,475 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0, startTime=1463530395474, containerId=container_1463530395095_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:13:15 Starting to run new task attempt: attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,476 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,477 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,477 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:15,478 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530395095_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:13:15,478 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:15,478 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530395095_0001_00_000001, Exiting
2016-05-18 00:13:15,478 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530395095_0001_00_000001
2016-05-18 00:13:15,478 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530395095_0001_00_000001
2016-05-18 00:13:15,479 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, VertexName: D27F6E9D4C064764BAA82EE0A9034C00, VertexParallelism: 1, TaskAttemptID:attempt_1463530395095_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D43FEC2C533440559297E3A2FCC21D83, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=CC7C82307BB441C4A2C97CA7B9F46328, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:15,479 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:15,480 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,480 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530395095_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:15,484 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530395095_0001_00_000001 completed successfully
2016-05-18 00:13:15,485 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530395095_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,485 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530395095_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:15,490 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:15,494 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:15,500 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D27F6E9D4C064764BAA82EE0A9034C00, ordinal: 1
2016-05-18 00:13:15,500 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D27F6E9D4C064764BAA82EE0A9034C00, mem on start (mb), free: 88, total: 390, max: 672
2016-05-18 00:13:15,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:15,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:15,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:15,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:15,501 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CC7C82307BB441C4A2C97CA7B9F46328:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [D43FEC2C533440559297E3A2FCC21D83:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:15,501 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:15,502 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:15,502 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:15,502 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CC7C82307BB441C4A2C97CA7B9F46328:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [D43FEC2C533440559297E3A2FCC21D83:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:15,503 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:15,503 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D43FEC2C533440559297E3A2FCC21D83 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:15,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:15,507 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:15,506 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D43FEC2C533440559297E3A2FCC21D83: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:15,508 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D43FEC2C533440559297E3A2FCC21D83: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:15,509 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D43FEC2C533440559297E3A2FCC21D83 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:15,510 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D43FEC2C533440559297E3A2FCC21D83
2016-05-18 00:13:15,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:15,511 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,512 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D43FEC2C533440559297E3A2FCC21D83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:15,514 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530395095_0001_1_00_000000_0_10015} to DISK_DIRECT, csize=1089, dsize=1085, EndTime=1463530395514, TimeTaken=2, Rate=0.52 MB/s
2016-05-18 00:13:15,515 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D43FEC2C533440559297E3A2FCC21D83
2016-05-18 00:13:15,515 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:15,515 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D43FEC2C533440559297E3A2FCC21D83, wasInterrupted=false
2016-05-18 00:13:15,516 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]]
2016-05-18 00:13:15,516 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:15,517 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:15,517 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_00_000000_0_10015/file.out, len=1089, isLocal=true
2016-05-18 00:13:15,517 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 1089 bytes from disk
2016-05-18 00:13:15,517 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:15,517 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:15,518 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] streamed: true, id: 901A9D51FB954F1FBFA546C978B3D5E7
2016-05-18 00:13:15,518 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-18 00:13:15,518 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D43FEC2C533440559297E3A2FCC21D83
2016-05-18 00:13:15,518 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D43FEC2C533440559297E3A2FCC21D83: Shuffle Runner thread complete
2016-05-18 00:13:15,518 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{?}:ALL]], id: CED45E3A480E4D7EB78C803E6CDC2817
2016-05-18 00:13:15,519 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CED45E3A480E4D7EB78C803E6CDC2817
2016-05-18 00:13:15,522 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CC7C82307BB441C4A2C97CA7B9F46328 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2e806493, comparator=cascading.tuple.hadoop.util.TupleComparator@6f4b06be, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:15,523 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CC7C82307BB441C4A2C97CA7B9F46328: , UsingHashComparator=false
2016-05-18 00:13:15,547 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:15,548 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:15,548 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CC7C82307BB441C4A2C97CA7B9F46328: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:15,548 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CC7C82307BB441C4A2C97CA7B9F46328: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:15,549 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(pearson)[by:[{1}:'l'][numSelfJoins:1]] 901A9D51FB954F1FBFA546C978B3D5E7, for 1 inputs
2016-05-18 00:13:15,550 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:15,550 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:15,550 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D27F6E9D4C064764BAA82EE0A9034C00, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:15,551 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D43FEC2C533440559297E3A2FCC21D83: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:15,584 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,584 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D27F6E9D4C064764BAA82EE0A9034C00, mem on close (mb), free: 164, total: 390, max: 672
2016-05-18 00:13:15,585 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D43FEC2C533440559297E3A2FCC21D83
2016-05-18 00:13:15,585 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D43FEC2C533440559297E3A2FCC21D83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:15,585 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CC7C82307BB441C4A2C97CA7B9F46328: Starting flush of map output
2016-05-18 00:13:15,585 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CC7C82307BB441C4A2C97CA7B9F46328: Span0.length = 174, perItem = 75
2016-05-18 00:13:15,601 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CC7C82307BB441C4A2C97CA7B9F46328: done sorting span=0, length=174, time=16
2016-05-18 00:13:15,601 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CC7C82307BB441C4A2C97CA7B9F46328: Heap = SpanIterator<0:173> (span=Span[16777216,13104]),
2016-05-18 00:13:15,602 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CC7C82307BB441C4A2C97CA7B9F46328: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_01_000000_0_10017_0/file.out
2016-05-18 00:13:15,610 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:15 Completed running task attempt: attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,611 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:13:15,611 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530395095_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:13:15,612 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530395095_0001_1_01_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530395551, Process_Duration=33, Process_End_Time=1463530395584, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7]]
2016-05-18 00:13:15,613 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530395095_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,614 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,615 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:15,613 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, taskAttemptId=attempt_1463530395095_0001_1_01_000000_0, creationTime=1463530395470, allocationTime=1463530395471, startTime=1463530395474, finishTime=1463530395613, timeTaken=139, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:15,615 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,615 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000002
2016-05-18 00:13:15,615 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530395095_0001_1_01_000000_0
2016-05-18 00:13:15,617 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_FINISHED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, taskId=task_1463530395095_0001_1_01_000000, startTime=1463530395474, finishTime=1463530395617, timeTaken=143, status=SUCCEEDED, successfulAttemptID=attempt_1463530395095_0001_1_01_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395551, Process_Duration=33, Process_End_Time=1463530395584, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7
2016-05-18 00:13:15,618 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:15,618 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,619 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_FINISHED]: vertexName=D27F6E9D4C064764BAA82EE0A9034C00, vertexId=vertex_1463530395095_0001_1_01, initRequestedTime=1463530395308, initedTime=1463530395319, startRequestedTime=1463530395365, startedTime=1463530395365, finishTime=1463530395618, timeTaken=253, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395551, Process_Duration=33, Process_End_Time=1463530395584, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=7, vertexStats=firstTaskStartTime=1463530395474, firstTasksToStart=[ task_1463530395095_0001_1_01_000000 ], lastTaskFinishTime=1463530395613, lastTasksToFinish=[ task_1463530395095_0001_1_01_000000 ], minTaskDuration=139, maxTaskDuration=139, avgTaskDuration=139.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530395095_0001_1_01_000000 ], longestDurationTasks=[ task_1463530395095_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:15,619 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:15,620 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530395095_0001_00_000002, stoppedTime=1463530395620, exitStatus=0
2016-05-18 00:13:15,620 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] attempt: attempt_1463530395095_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:15,620 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:15,620 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:15,620 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CC7C82307BB441C4A2C97CA7B9F46328
2016-05-18 00:13:15,622 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530395095_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:13:15,622 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D27F6E9D4C064764BAA82EE0A9034C00, destinationVertexName=CC7C82307BB441C4A2C97CA7B9F46328. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:15,622 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,623 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_STARTED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, taskId=task_1463530395095_0001_1_02_000000, scheduledTime=1463530395623, launchTime=1463530395623
2016-05-18 00:13:15,623 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,626 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530395095_0001_00_000003, launchTime=1463530395625
2016-05-18 00:13:15,626 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000003
2016-05-18 00:13:15,627 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530395095_0001_1_02_000000_0] started. Is using containerId: [container_1463530395095_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 00:13:15,628 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0, startTime=1463530395627, containerId=container_1463530395095_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 00:13:15,630 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530395095_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:13:15,630 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:15,630 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530395095_0001_00_000002, Exiting
2016-05-18 00:13:15,630 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530395095_0001_00_000002
2016-05-18 00:13:15,631 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530395095_0001_00_000002
2016-05-18 00:13:15,631 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530395095_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:15,631 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530395095_0001_00_000002 completed successfully
2016-05-18 00:13:15,631 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15 Starting to run new task attempt: attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,632 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,632 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,632 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:15,632 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, VertexName: CC7C82307BB441C4A2C97CA7B9F46328, VertexParallelism: 1, TaskAttemptID:attempt_1463530395095_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D27F6E9D4C064764BAA82EE0A9034C00, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A7D1737F6E25420CB295B88F11AFAFE6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:15,632 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:15,633 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,633 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530395095_0001_1_02_000000_0 1 events [0,1) total 1 vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,634 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530395095_0001_1_02_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:15,639 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:15,639 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:15,650 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CC7C82307BB441C4A2C97CA7B9F46328, ordinal: 2
2016-05-18 00:13:15,650 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC7C82307BB441C4A2C97CA7B9F46328, mem on start (mb), free: 157, total: 390, max: 672
2016-05-18 00:13:15,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:15,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:15,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:15,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:15,651 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A7D1737F6E25420CB295B88F11AFAFE6:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [D27F6E9D4C064764BAA82EE0A9034C00:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:15,652 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:15,652 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:15,653 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:15,653 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A7D1737F6E25420CB295B88F11AFAFE6:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [D27F6E9D4C064764BAA82EE0A9034C00:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:15,653 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:15,654 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D27F6E9D4C064764BAA82EE0A9034C00 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:15,654 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:15,654 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:15,658 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D27F6E9D4C064764BAA82EE0A9034C00: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:15,658 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D27F6E9D4C064764BAA82EE0A9034C00: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:15,659 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D27F6E9D4C064764BAA82EE0A9034C00 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:15,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D27F6E9D4C064764BAA82EE0A9034C00
2016-05-18 00:13:15,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:15,662 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,663 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D27F6E9D4C064764BAA82EE0A9034C00: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:15,664 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530395095_0001_1_01_000000_0_10017} to DISK_DIRECT, csize=7519, dsize=7515, EndTime=1463530395664, TimeTaken=1, Rate=7.17 MB/s
2016-05-18 00:13:15,664 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D27F6E9D4C064764BAA82EE0A9034C00
2016-05-18 00:13:15,664 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:15,664 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D27F6E9D4C064764BAA82EE0A9034C00, wasInterrupted=false
2016-05-18 00:13:15,666 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:15,667 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:15,668 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{?}:ALL]]
2016-05-18 00:13:15,669 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_01_000000_0_10017/file.out, len=7519, isLocal=true
2016-05-18 00:13:15,669 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 7519 bytes from disk
2016-05-18 00:13:15,669 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:15,670 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{?}:ALL]] streamed: true, id: CED45E3A480E4D7EB78C803E6CDC2817
2016-05-18 00:13:15,670 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(pearson)[by:[{2}:'n1', 'n2']], id: AD8B24C3797D44FFA9C2110FF2D2CC88
2016-05-18 00:13:15,670 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:15,670 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] AD8B24C3797D44FFA9C2110FF2D2CC88
2016-05-18 00:13:15,671 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-18 00:13:15,672 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D27F6E9D4C064764BAA82EE0A9034C00
2016-05-18 00:13:15,672 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D27F6E9D4C064764BAA82EE0A9034C00: Shuffle Runner thread complete
2016-05-18 00:13:15,673 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A7D1737F6E25420CB295B88F11AFAFE6 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6f63c721, comparator=cascading.tuple.hadoop.util.TupleComparator@7fa680a8, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:15,673 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A7D1737F6E25420CB295B88F11AFAFE6: , UsingHashComparator=false
2016-05-18 00:13:15,686 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:15,687 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:15,687 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A7D1737F6E25420CB295B88F11AFAFE6: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:15,687 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A7D1737F6E25420CB295B88F11AFAFE6: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:15,687 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{?}:ALL]] CED45E3A480E4D7EB78C803E6CDC2817, for 1 inputs
2016-05-18 00:13:15,687 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CC7C82307BB441C4A2C97CA7B9F46328, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:15,688 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D27F6E9D4C064764BAA82EE0A9034C00: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:15,709 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,709 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC7C82307BB441C4A2C97CA7B9F46328, mem on close (mb), free: 53, total: 390, max: 672
2016-05-18 00:13:15,710 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D27F6E9D4C064764BAA82EE0A9034C00
2016-05-18 00:13:15,710 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D27F6E9D4C064764BAA82EE0A9034C00: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:15,710 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A7D1737F6E25420CB295B88F11AFAFE6: Starting flush of map output
2016-05-18 00:13:15,710 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A7D1737F6E25420CB295B88F11AFAFE6: Span0.length = 87, perItem = 72
2016-05-18 00:13:15,712 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A7D1737F6E25420CB295B88F11AFAFE6: done sorting span=0, length=87, time=1
2016-05-18 00:13:15,713 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A7D1737F6E25420CB295B88F11AFAFE6: Heap = SpanIterator<0:86> (span=Span[16777216,6291]),
2016-05-18 00:13:15,714 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A7D1737F6E25420CB295B88F11AFAFE6: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_02_000000_0_10021_0/file.out
2016-05-18 00:13:15,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:15 Completed running task attempt: attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,719 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0, askedToStop=false
2016-05-18 00:13:15,719 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530395095_0001_1_02_000000_0, stopRequested=false
2016-05-18 00:13:15,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530395095_0001_1_02_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530395687, Process_Duration=22, Process_End_Time=1463530395709, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2]]
2016-05-18 00:13:15,721 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530395095_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,721 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,721 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:15,721 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,721 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, taskAttemptId=attempt_1463530395095_0001_1_02_000000_0, creationTime=1463530395623, allocationTime=1463530395625, startTime=1463530395627, finishTime=1463530395721, timeTaken=94, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:15,722 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530395095_0001_1_02_000000_0
2016-05-18 00:13:15,722 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000003
2016-05-18 00:13:15,723 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_FINISHED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, taskId=task_1463530395095_0001_1_02_000000, startTime=1463530395627, finishTime=1463530395723, timeTaken=96, status=SUCCEEDED, successfulAttemptID=attempt_1463530395095_0001_1_02_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395687, Process_Duration=22, Process_End_Time=1463530395709, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2
2016-05-18 00:13:15,724 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:15,724 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_FINISHED]: vertexName=CC7C82307BB441C4A2C97CA7B9F46328, vertexId=vertex_1463530395095_0001_1_02, initRequestedTime=1463530395320, initedTime=1463530395332, startRequestedTime=1463530395368, startedTime=1463530395372, finishTime=1463530395725, timeTaken=353, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=6291, OUTPUT_BYTES_WITH_OVERHEAD=4340, OUTPUT_BYTES_PHYSICAL=4344, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395687, Process_Duration=22, Process_End_Time=1463530395709, Read_Duration=11, Tuples_Read=174, Tuples_Written=87, Write_Duration=2, vertexStats=firstTaskStartTime=1463530395627, firstTasksToStart=[ task_1463530395095_0001_1_02_000000 ], lastTaskFinishTime=1463530395721, lastTasksToFinish=[ task_1463530395095_0001_1_02_000000 ], minTaskDuration=94, maxTaskDuration=94, avgTaskDuration=94.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530395095_0001_1_02_000000 ], longestDurationTasks=[ task_1463530395095_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:15,726 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:15,727 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530395095_0001_00_000003, stoppedTime=1463530395727, exitStatus=0
2016-05-18 00:13:15,727 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] attempt: attempt_1463530395095_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:15,727 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A7D1737F6E25420CB295B88F11AFAFE6
2016-05-18 00:13:15,727 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CC7C82307BB441C4A2C97CA7B9F46328, destinationVertexName=A7D1737F6E25420CB295B88F11AFAFE6. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:15,728 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,728 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:15,728 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:15,728 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530395095_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 00:13:15,728 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_STARTED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, taskId=task_1463530395095_0001_1_03_000000, scheduledTime=1463530395728, launchTime=1463530395728
2016-05-18 00:13:15,729 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530395095_0001_00_000004, launchTime=1463530395730
2016-05-18 00:13:15,730 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000004
2016-05-18 00:13:15,731 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530395095_0001_1_03_000000_0] started. Is using containerId: [container_1463530395095_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-18 00:13:15,731 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0, startTime=1463530395731, containerId=container_1463530395095_0001_00_000004, nodeId=127.0.0.1:0
2016-05-18 00:13:15,732 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15 Starting to run new task attempt: attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,732 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,732 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,732 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:15,732 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, VertexName: A7D1737F6E25420CB295B88F11AFAFE6, VertexParallelism: 1, TaskAttemptID:attempt_1463530395095_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CC7C82307BB441C4A2C97CA7B9F46328, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=5990AD4214364C3C8BDD323947B28F35, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:13:15,733 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:15,733 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,737 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530395095_0001_1_03_000000_0 1 events [0,1) total 1 vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,738 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530395095_0001_1_03_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:15,741 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530395095_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 00:13:15,741 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000003: 17 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:15,741 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530395095_0001_00_000003, Exiting
2016-05-18 00:13:15,741 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530395095_0001_00_000003
2016-05-18 00:13:15,741 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530395095_0001_00_000003
2016-05-18 00:13:15,742 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530395095_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:15,742 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530395095_0001_00_000003 completed successfully
2016-05-18 00:13:15,744 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:15,746 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:15,755 INFO output.MROutput (MROutput.java:initialize(421)) - 5990AD4214364C3C8BDD323947B28F35: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:13:15,761 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A7D1737F6E25420CB295B88F11AFAFE6, ordinal: 3
2016-05-18 00:13:15,761 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7D1737F6E25420CB295B88F11AFAFE6, mem on start (mb), free: 48, total: 390, max: 672
2016-05-18 00:13:15,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:15,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:15,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:15,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:15,763 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[5990AD4214364C3C8BDD323947B28F35:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [CC7C82307BB441C4A2C97CA7B9F46328:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:15,763 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:15,763 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:15,764 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:13:15,764 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[5990AD4214364C3C8BDD323947B28F35:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [CC7C82307BB441C4A2C97CA7B9F46328:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:15,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:15,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CC7C82307BB441C4A2C97CA7B9F46328 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:15,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:15,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:15,765 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CC7C82307BB441C4A2C97CA7B9F46328: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:15,765 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CC7C82307BB441C4A2C97CA7B9F46328: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:15,767 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CC7C82307BB441C4A2C97CA7B9F46328 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:15,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CC7C82307BB441C4A2C97CA7B9F46328
2016-05-18 00:13:15,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:15,769 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,770 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC7C82307BB441C4A2C97CA7B9F46328: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:15,782 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530395095_0001_1_02_000000_0_10021} to DISK_DIRECT, csize=4344, dsize=4340, EndTime=1463530395782, TimeTaken=2, Rate=2.07 MB/s
2016-05-18 00:13:15,783 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CC7C82307BB441C4A2C97CA7B9F46328
2016-05-18 00:13:15,783 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(pearson)[by:[{2}:'n1', 'n2']]
2016-05-18 00:13:15,783 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:15,783 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CC7C82307BB441C4A2C97CA7B9F46328, wasInterrupted=false
2016-05-18 00:13:15,784 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:15,785 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:15,786 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(pearson)[by:[{2}:'n1', 'n2']] streamed: true, id: AD8B24C3797D44FFA9C2110FF2D2CC88
2016-05-18 00:13:15,786 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"], id: 5990AD4214364C3C8BDD323947B28F35
2016-05-18 00:13:15,787 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite"]
2016-05-18 00:13:15,788 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(pearson)[by:[{2}:'n1', 'n2']] AD8B24C3797D44FFA9C2110FF2D2CC88, for 1 inputs
2016-05-18 00:13:15,788 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001_wd/localmode-local-dir/output/attempt_1463530395095_0001_1_02_000000_0_10021/file.out, len=4344, isLocal=true
2016-05-18 00:13:15,789 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 4344 bytes from disk
2016-05-18 00:13:15,789 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:15,789 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:15,789 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 4301 bytes
2016-05-18 00:13:15,790 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CC7C82307BB441C4A2C97CA7B9F46328
2016-05-18 00:13:15,790 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CC7C82307BB441C4A2C97CA7B9F46328: Shuffle Runner thread complete
2016-05-18 00:13:15,790 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A7D1737F6E25420CB295B88F11AFAFE6, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:13:15,791 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CC7C82307BB441C4A2C97CA7B9F46328: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:15,802 INFO output.MROutput (MROutput.java:close(526)) - 5990AD4214364C3C8BDD323947B28F35 closed
2016-05-18 00:13:15,803 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463530395095_0001_1_03_000000_0 given a go for committing the task output.
2016-05-18 00:13:15,804 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,804 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7D1737F6E25420CB295B88F11AFAFE6, mem on close (mb), free: 263, total: 390, max: 672
2016-05-18 00:13:15,804 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CC7C82307BB441C4A2C97CA7B9F46328
2016-05-18 00:13:15,804 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC7C82307BB441C4A2C97CA7B9F46328: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:15,805 INFO output.MROutput (MROutput.java:close(526)) - 5990AD4214364C3C8BDD323947B28F35 closed
2016-05-18 00:13:15,805 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:15 Completed running task attempt: attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,805 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0, askedToStop=false
2016-05-18 00:13:15,805 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530395095_0001_1_03_000000_0, stopRequested=false
2016-05-18 00:13:15,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530395095_0001_1_03_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=18, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530395790, Process_Duration=14, Process_End_Time=1463530395804, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-18 00:13:15,808 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530395095_0001_1_03_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,808 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:15,808 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:15,808 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, taskAttemptId=attempt_1463530395095_0001_1_03_000000_0, creationTime=1463530395729, allocationTime=1463530395729, startTime=1463530395731, finishTime=1463530395808, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:15,809 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530395095_0001_00_000004
2016-05-18 00:13:15,809 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530395095_0001_1_03_000000_0
2016-05-18 00:13:15,810 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:TASK_FINISHED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, taskId=task_1463530395095_0001_1_03_000000, startTime=1463530395731, finishTime=1463530395809, timeTaken=78, status=SUCCEEDED, successfulAttemptID=attempt_1463530395095_0001_1_03_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=18, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395790, Process_Duration=14, Process_End_Time=1463530395804, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-18 00:13:15,810 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:15,811 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6]
2016-05-18 00:13:15,812 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:VERTEX_FINISHED]: vertexName=A7D1737F6E25420CB295B88F11AFAFE6, vertexId=vertex_1463530395095_0001_1_03, initRequestedTime=1463530395333, initedTime=1463530395342, startRequestedTime=1463530395375, startedTime=1463530395376, finishTime=1463530395811, timeTaken=435, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4344, SHUFFLE_BYTES=4344, SHUFFLE_BYTES_DECOMPRESSED=4340, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=4344, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=18, MERGE_PHASE_TIME=25, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530395790, Process_Duration=14, Process_End_Time=1463530395804, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463530395731, firstTasksToStart=[ task_1463530395095_0001_1_03_000000 ], lastTaskFinishTime=1463530395808, lastTasksToFinish=[ task_1463530395095_0001_1_03_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530395095_0001_1_03_000000 ], longestDurationTasks=[ task_1463530395095_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:15,812 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:15,813 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530395095_0001_00_000004, stoppedTime=1463530395813, exitStatus=0
2016-05-18 00:13:15,813 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:15,813 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:15,813 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530395095_0001_1_02 [CC7C82307BB441C4A2C97CA7B9F46328]
2016-05-18 00:13:15,813 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530395095_0001_1_00 [D43FEC2C533440559297E3A2FCC21D83]
2016-05-18 00:13:15,813 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530395095_0001_1_01 [D27F6E9D4C064764BAA82EE0A9034C00]
2016-05-18 00:13:15,814 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463530395095_0001_1
2016-05-18 00:13:15,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463530395095_0001_1
2016-05-18 00:13:15,814 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 5990AD4214364C3C8BDD323947B28F35 for vertex: vertex_1463530395095_0001_1_03 [A7D1737F6E25420CB295B88F11AFAFE6], outputName: 5990AD4214364C3C8BDD323947B28F35
2016-05-18 00:13:15,816 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530395095_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:13:15,816 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530395095_0001_00_000004. Relying on regular task shutdown for it to end
2016-05-18 00:13:15,820 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530395095_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-18 00:13:15,820 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530395095_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:15,820 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530395095_0001_00_000004, Exiting
2016-05-18 00:13:15,820 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530395095_0001_00_000004
2016-05-18 00:13:15,820 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530395095_0001_00_000004
2016-05-18 00:13:15,821 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530395095_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:15,821 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530395095_0001_00_000004 completed successfully
2016-05-18 00:13:15,824 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:5990AD4214364C3C8BDD323947B28F35 of vertex/vertexGroup:A7D1737F6E25420CB295B88F11AFAFE6 isVertexGroupOutput:false
2016-05-18 00:13:15,824 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:15,828 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463530395095_0001_1, queueSize=0
2016-05-18 00:13:15,828 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530395095_0001_1][Event:DAG_FINISHED]: dagId=dag_1463530395095_0001_1, startTime=1463530395305, finishTime=1463530395824, timeTaken=519, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=980, AM_GC_TIME_MILLIS=21, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=114, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21071, OUTPUT_BYTES_WITH_OVERHEAD=12940, OUTPUT_BYTES_PHYSICAL=12952, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=12952, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=3, SHUFFLE_BYTES=12952, SHUFFLE_BYTES_DECOMPRESSED=12940, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=12952, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=49, FIRST_EVENT_RECEIVED=14, LAST_EVENT_RECEIVED=14, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854121582472, Process_Duration=74, Process_End_Time=5854121582546, Read_Duration=20, Tuples_Read=303, Tuples_Written=317, Write_Duration=11, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-18 00:13:15,829 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463530395095_0001_1 finished with state: SUCCEEDED
2016-05-18 00:13:15,829 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530395095_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:13:15 Completed Dag: dag_1463530395095_0001_1
2016-05-18 00:13:15,829 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463530395095_0001_1, dagState=SUCCEEDED
2016-05-18 00:13:15,829 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530395095_0001_1_post
2016-05-18 00:13:15,829 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:15,830 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:13:15,830 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:13:15,830 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, with id=dag_1463530395095_0001_1
2016-05-18 00:13:15,830 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E] pearson/(1/1) ...mposite/pearson/composite, with id=dag_1463530395095_0001_1
2016-05-18 00:13:15,840 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mposite/pearson/composite, applicationId=application_1463530395095_0001
2016-05-18 00:13:15,842 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:13:15,843 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:13:15,843 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:13:15,843 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:13:15,843 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:13:15,843 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:13:15,844 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:13:15,844 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:13:15,844 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:13:15,844 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:13:15,845 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:13:15,845 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:13:15,846 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:13:15,847 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:13:15,850 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/63425FD1DAA8470C9CC9709A78BD5DBF/E768CD59B6BA4261B3D5DF15030EBE2E/.tez/application_1463530395095_0001
2016-05-18 00:13:15,852 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite/_temporary
2016-05-18 00:13:15,852 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [pearson] completed in: 00:00.787
2016-05-18 00:13:15,942 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testPearsonDistanceComposite/pearson/composite
2016-05-18 00:13:15,980 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:13:15,981 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:13:16,062 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.050
2016-05-18 00:13:16,106 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.094
2016-05-18 00:13:16,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:16,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:16,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:13:16,142 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt
2016-05-18 00:13:16,174 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-05-18 00:13:16,343 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-18 00:13:16,343 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:16,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-05-18 00:13:16,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-18 00:13:16,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-18 00:13:16,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-18 00:13:16,345 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...anDistance/euclidean/long
2016-05-18 00:13:16,345 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:13:16,362 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-18 00:13:16,366 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:16,367 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:13:16,367 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:13:16,376 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001 doesn't exist and is created
2016-05-18 00:13:16,398 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:13:16,398 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:16,399 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd
2016-05-18 00:13:16,410 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463530396367_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:16,410 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:16,411 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:16,412 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:13:16,412 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:13:16,412 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:13:16,413 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:13:16,414 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:13:16,415 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:13:16,415 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:13:16,416 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:13:16,416 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:13:16,416 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:13:16,416 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:13:16,416 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:13:16,417 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:13:16,417 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:16,417 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:16,418 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:13:16,418 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:13:16,420 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:13:16,421 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:13:16,421 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:13:16,421 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-log-dir
2016-05-18 00:13:16,422 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-log-dir/history.txt.appattempt_1463530396367_0001_000000, maxErrors=10
2016-05-18 00:13:16,422 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:13:16,422 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463530396367_0001_000000, appSubmitTime=1463530396410, launchTime=1463530396410
2016-05-18 00:13:16,432 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:13:16,433 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:45270
2016-05-18 00:13:16,437 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463530396367
2016-05-18 00:13:16,440 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463530396367_0001_000000, startTime=1463530396440
2016-05-18 00:13:16,440 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:13:16,499 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:13:16,499 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:13:16,499 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463530396367_0001, dagName=[791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-18 00:13:16,519 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-18 00:13:16,519 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:13:16,521 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463530396367_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-log-dir/dag_1463530396367_0001_1.dot
2016-05-18 00:13:16,523 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-log-dir/dag_1463530396367_0001_1-tez-dag.pb.txt
2016-05-18 00:13:16,536 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530396367_0001_1
2016-05-18 00:13:16,537 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,537 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-18 00:13:16 Running Dag: dag_1463530396367_0001_1
2016-05-18 00:13:16,547 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463530396367_0001_1, submitTime=1463530396519
2016-05-18 00:13:16,548 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001/tez-conf.pb" } size: 36897 timestamp: 1463530396000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:16,548 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,549 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:16,549 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001/tez-conf.pb" } size: 36897 timestamp: 1463530396000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:16,549 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:16,550 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001/tez-conf.pb" } size: 36897 timestamp: 1463530396000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:16,550 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:16,551 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001/tez-conf.pb" } size: 36897 timestamp: 1463530396000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:16,551 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,551 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:16,552 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:13:16,552 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463530396367_0001_1, initTime=1463530396552
2016-05-18 00:13:16,552 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530396367_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:13:16,553 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:13:16,553 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:DAG_STARTED]: dagID=dag_1463530396367_0001_1, startTime=1463530396553
2016-05-18 00:13:16,553 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530396367_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:13:16,554 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F0156A659DDA4644B0229FE289F097BA : {BFDA348C69604D059B1109BFFDC2A16E={InputName=BFDA348C69604D059B1109BFFDC2A16E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:13:16,554 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BFDA348C69604D059B1109BFFDC2A16E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:13:16,554 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,554 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,554 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,555 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,555 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463530396367_0001, dagName=[791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long
2016-05-18 00:13:16,555 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:13:16,555 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,558 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BFDA348C69604D059B1109BFFDC2A16E on vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,560 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:16,560 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,561 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,561 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530396367_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:16,562 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, vertexId=vertex_1463530396367_0001_1_01, initRequestedTime=1463530396555, initedTime=1463530396562, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:16,562 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:16,562 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,561 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BFDA348C69604D059B1109BFFDC2A16E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:13:16,568 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:16,568 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,568 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,569 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530396367_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:16,569 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, vertexId=vertex_1463530396367_0001_1_02, initRequestedTime=1463530396562, initedTime=1463530396569, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:16,569 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:16,570 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,572 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:16,573 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:13:16,573 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 828 Min splitLength: 52428800 New desired splits: 1 Total length: 828 Original splits: 2
2016-05-18 00:13:16,573 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 828 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 828 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:13:16,574 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:13:16,573 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,574 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BFDA348C69604D059B1109BFFDC2A16E on vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,574 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530396367_0001_1_03, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:16,575 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3], numAdditionalOutputs=1
2016-05-18 00:13:16,576 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=26AF33801AB34C4D8330E7709122DD7D, vertex=vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:13:16,577 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D1DB9DFA72364EF487BCB83C38360FD3:26AF33801AB34C4D8330E7709122DD7D using oldmapred API
2016-05-18 00:13:16,578 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=26AF33801AB34C4D8330E7709122DD7D, vertexName=D1DB9DFA72364EF487BCB83C38360FD3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:13:16,579 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, vertexId=vertex_1463530396367_0001_1_03, initRequestedTime=1463530396570, initedTime=1463530396575, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:16,579 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:16,580 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:16,582 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BFDA348C69604D059B1109BFFDC2A16E=forAllWorkUnits=true, update=[1]}
2016-05-18 00:13:16,582 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] parallelism set to 1
2016-05-18 00:13:16,583 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:16,583 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463530396367_0001, with dag id: dag_1463530396367_0001_1
2016-05-18 00:13:16,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F0156A659DDA4644B0229FE289F097BA, vertexId=vertex_1463530396367_0001_1_00, initRequestedTime=1463530396554, initedTime=1463530396584, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:13:16,585 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] with distanceFromRoot: 0
2016-05-18 00:13:16,585 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:13:16,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530396367_0001_1_00, startRequestedTime=1463530396555, startedTime=1463530396586
2016-05-18 00:13:16,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530396367_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:13:16,587 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:16,587 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530396367_0001_1_00 for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] numStartedSources: 1 numSources: 1
2016-05-18 00:13:16,587 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] with distanceFromRoot: 1
2016-05-18 00:13:16,587 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F0156A659DDA4644B0229FE289F097BA
2016-05-18 00:13:16,588 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,591 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530396367_0001_1_01, startRequestedTime=1463530396587, startedTime=1463530396588
2016-05-18 00:13:16,591 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:16,592 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530396367_0001_1_01 for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] numStartedSources: 1 numSources: 1
2016-05-18 00:13:16,592 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] with distanceFromRoot: 2
2016-05-18 00:13:16,593 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BA83B71CC23A4663B2B3DC10D2BC90BC with 0 source tasks and 1 pending tasks
2016-05-18 00:13:16,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_STARTED]: vertexName=F0156A659DDA4644B0229FE289F097BA, taskId=task_1463530396367_0001_1_00_000000, scheduledTime=1463530396592, launchTime=1463530396592
2016-05-18 00:13:16,594 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F0156A659DDA4644B0229FE289F097BA in vertex: BA83B71CC23A4663B2B3DC10D2BC90BC numBipartiteSourceTasks: 1
2016-05-18 00:13:16,595 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530396367_0001_1_02, startRequestedTime=1463530396592, startedTime=1463530396595
2016-05-18 00:13:16,596 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:16,596 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,597 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530396367_0001_1_02 for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] numStartedSources: 1 numSources: 1
2016-05-18 00:13:16,597 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A0B199F003B143BC87E60C8C3E83A9CD with 0 source tasks and 1 pending tasks
2016-05-18 00:13:16,597 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] with distanceFromRoot: 3
2016-05-18 00:13:16,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530396367_0001_1_03, startRequestedTime=1463530396597, startedTime=1463530396598
2016-05-18 00:13:16,599 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:16,599 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BA83B71CC23A4663B2B3DC10D2BC90BC in vertex: A0B199F003B143BC87E60C8C3E83A9CD numBipartiteSourceTasks: 1
2016-05-18 00:13:16,600 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D1DB9DFA72364EF487BCB83C38360FD3 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:16,601 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:13:16,601 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A0B199F003B143BC87E60C8C3E83A9CD in vertex: D1DB9DFA72364EF487BCB83C38360FD3 numBipartiteSourceTasks: 1
2016-05-18 00:13:16,604 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530396367_0001_00_000001, launchTime=1463530396603
2016-05-18 00:13:16,604 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000001
2016-05-18 00:13:16,605 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530396367_0001_1_00_000000_0] started. Is using containerId: [container_1463530396367_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:13:16,605 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F0156A659DDA4644B0229FE289F097BA, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0, startTime=1463530396605, containerId=container_1463530396367_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:13:16,606 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16 Starting to run new task attempt: attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,606 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,607 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,607 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:16,607 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: F0156A659DDA4644B0229FE289F097BA, VertexParallelism: 1, TaskAttemptID:attempt_1463530396367_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BFDA348C69604D059B1109BFFDC2A16E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:16,607 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:16,608 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,613 INFO input.MRInput (MRInput.java:initialize(440)) - BFDA348C69604D059B1109BFFDC2A16E using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:13:16,613 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BFDA348C69604D059B1109BFFDC2A16E
2016-05-18 00:13:16,613 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:16,615 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463530396367_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:13:16,616 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530396367_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:13:16,625 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F0156A659DDA4644B0229FE289F097BA, ordinal: 0
2016-05-18 00:13:16,625 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F0156A659DDA4644B0229FE289F097BA, mem on start (mb), free: 109, total: 390, max: 672
2016-05-18 00:13:16,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:16,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:16,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:16,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:16,629 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BFDA348C69604D059B1109BFFDC2A16E:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BA83B71CC23A4663B2B3DC10D2BC90BC:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:16,629 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:16,629 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:16,630 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:13:16,630 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BFDA348C69604D059B1109BFFDC2A16E:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BA83B71CC23A4663B2B3DC10D2BC90BC:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:16,630 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:16,630 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BFDA348C69604D059B1109BFFDC2A16E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:16,631 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BFDA348C69604D059B1109BFFDC2A16E
2016-05-18 00:13:16,631 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:16,631 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:16,632 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:16,632 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,634 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:0+414, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:414+414], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=828}
2016-05-18 00:13:16,634 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BFDA348C69604D059B1109BFFDC2A16E initialized RecordReader from event
2016-05-18 00:13:16,636 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:16,636 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"] streamed: true, id: BFDA348C69604D059B1109BFFDC2A16E
2016-05-18 00:13:16,636 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: AAB7D6DE35B94AFA9FD9FA3B335E1DA0
2016-05-18 00:13:16,637 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] AAB7D6DE35B94AFA9FD9FA3B335E1DA0
2016-05-18 00:13:16,638 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BA83B71CC23A4663B2B3DC10D2BC90BC using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@69557cc5, comparator=cascading.tuple.hadoop.util.TupleComparator@6b729f89, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:16,638 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BA83B71CC23A4663B2B3DC10D2BC90BC: , UsingHashComparator=false
2016-05-18 00:13:16,664 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:16,664 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:16,664 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BA83B71CC23A4663B2B3DC10D2BC90BC: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:16,664 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BA83B71CC23A4663B2B3DC10D2BC90BC: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:16,665 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:16,665 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F0156A659DDA4644B0229FE289F097BA, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:16,669 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,669 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F0156A659DDA4644B0229FE289F097BA, mem on close (mb), free: 161, total: 391, max: 672
2016-05-18 00:13:16,669 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Starting flush of map output
2016-05-18 00:13:16,669 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Span0.length = 35, perItem = 47
2016-05-18 00:13:16,670 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BA83B71CC23A4663B2B3DC10D2BC90BC: done sorting span=0, length=35, time=0
2016-05-18 00:13:16,670 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Heap = SpanIterator<0:34> (span=Span[16777216,1676]),
2016-05-18 00:13:16,672 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_00_000000_0_10027_0/file.out
2016-05-18 00:13:16,676 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:16 Completed running task attempt: attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,677 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:13:16,677 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530396367_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:13:16,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530396367_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463530396665, Process_Duration=4, Process_End_Time=1463530396669, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-18 00:13:16,678 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530396367_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,679 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,679 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:16,679 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F0156A659DDA4644B0229FE289F097BA, taskAttemptId=attempt_1463530396367_0001_1_00_000000_0, creationTime=1463530396595, allocationTime=1463530396602, startTime=1463530396605, finishTime=1463530396679, timeTaken=74, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:16,679 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530396367_0001_1_00_000000_0
2016-05-18 00:13:16,680 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000001
2016-05-18 00:13:16,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_FINISHED]: vertexName=F0156A659DDA4644B0229FE289F097BA, taskId=task_1463530396367_0001_1_00_000000, startTime=1463530396605, finishTime=1463530396680, timeTaken=75, status=SUCCEEDED, successfulAttemptID=attempt_1463530396367_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530396665, Process_Duration=4, Process_End_Time=1463530396669, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7
2016-05-18 00:13:16,681 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:16,681 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_FINISHED]: vertexName=F0156A659DDA4644B0229FE289F097BA, vertexId=vertex_1463530396367_0001_1_00, initRequestedTime=1463530396554, initedTime=1463530396584, startRequestedTime=1463530396555, startedTime=1463530396586, finishTime=1463530396681, timeTaken=95, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530396665, Process_Duration=4, Process_End_Time=1463530396669, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463530396605, firstTasksToStart=[ task_1463530396367_0001_1_00_000000 ], lastTaskFinishTime=1463530396679, lastTasksToFinish=[ task_1463530396367_0001_1_00_000000 ], minTaskDuration=74, maxTaskDuration=74, avgTaskDuration=74.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530396367_0001_1_00_000000 ], longestDurationTasks=[ task_1463530396367_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:16,682 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:16,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530396367_0001_00_000001, stoppedTime=1463530396683, exitStatus=0
2016-05-18 00:13:16,683 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] attempt: attempt_1463530396367_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:16,683 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:16,683 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:16,684 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BA83B71CC23A4663B2B3DC10D2BC90BC
2016-05-18 00:13:16,684 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F0156A659DDA4644B0229FE289F097BA, destinationVertexName=BA83B71CC23A4663B2B3DC10D2BC90BC. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:16,684 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,684 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530396367_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:13:16,685 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_STARTED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, taskId=task_1463530396367_0001_1_01_000000, scheduledTime=1463530396685, launchTime=1463530396685
2016-05-18 00:13:16,685 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,690 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530396367_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:13:16,691 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:16,691 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530396367_0001_00_000001, Exiting
2016-05-18 00:13:16,692 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530396367_0001_00_000001
2016-05-18 00:13:16,692 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530396367_0001_00_000001
2016-05-18 00:13:16,693 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530396367_0001_00_000002, launchTime=1463530396693
2016-05-18 00:13:16,694 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530396367_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:16,695 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530396367_0001_00_000001 completed successfully
2016-05-18 00:13:16,695 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000002
2016-05-18 00:13:16,696 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530396367_0001_1_01_000000_0] started. Is using containerId: [container_1463530396367_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:13:16,697 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0, startTime=1463530396696, containerId=container_1463530396367_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:13:16,699 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16 Starting to run new task attempt: attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,699 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,699 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,699 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:16,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: BA83B71CC23A4663B2B3DC10D2BC90BC, VertexParallelism: 1, TaskAttemptID:attempt_1463530396367_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F0156A659DDA4644B0229FE289F097BA, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A0B199F003B143BC87E60C8C3E83A9CD, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:16,700 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:16,702 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,702 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530396367_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,703 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530396367_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:16,706 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:16,707 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:16,714 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BA83B71CC23A4663B2B3DC10D2BC90BC, ordinal: 1
2016-05-18 00:13:16,714 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BA83B71CC23A4663B2B3DC10D2BC90BC, mem on start (mb), free: 156, total: 391, max: 672
2016-05-18 00:13:16,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:16,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:16,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:16,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:16,715 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F0156A659DDA4644B0229FE289F097BA:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [A0B199F003B143BC87E60C8C3E83A9CD:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:16,715 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:16,716 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:16,716 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:16,716 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F0156A659DDA4644B0229FE289F097BA:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [A0B199F003B143BC87E60C8C3E83A9CD:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:16,716 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:16,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F0156A659DDA4644B0229FE289F097BA being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:16,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:16,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:16,718 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F0156A659DDA4644B0229FE289F097BA: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:16,719 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F0156A659DDA4644B0229FE289F097BA: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:16,722 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F0156A659DDA4644B0229FE289F097BA with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:16,723 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F0156A659DDA4644B0229FE289F097BA
2016-05-18 00:13:16,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:16,725 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,728 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F0156A659DDA4644B0229FE289F097BA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:16,731 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-05-18 00:13:16,731 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530396367_0001_1_00_000000_0_10027} to DISK_DIRECT, csize=1089, dsize=1085, EndTime=1463530396731, TimeTaken=2, Rate=0.52 MB/s
2016-05-18 00:13:16,732 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F0156A659DDA4644B0229FE289F097BA
2016-05-18 00:13:16,732 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: AAB7D6DE35B94AFA9FD9FA3B335E1DA0
2016-05-18 00:13:16,733 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: D8912D8814E049A48866842D95AD58A1
2016-05-18 00:13:16,733 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] D8912D8814E049A48866842D95AD58A1
2016-05-18 00:13:16,732 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:16,734 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F0156A659DDA4644B0229FE289F097BA, wasInterrupted=false
2016-05-18 00:13:16,735 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:16,735 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:16,736 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_00_000000_0_10027/file.out, len=1089, isLocal=true
2016-05-18 00:13:16,736 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 1089 bytes from disk
2016-05-18 00:13:16,736 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:16,736 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:16,737 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-18 00:13:16,737 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F0156A659DDA4644B0229FE289F097BA
2016-05-18 00:13:16,737 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F0156A659DDA4644B0229FE289F097BA: Shuffle Runner thread complete
2016-05-18 00:13:16,738 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A0B199F003B143BC87E60C8C3E83A9CD using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1adde645, comparator=cascading.tuple.hadoop.util.TupleComparator@4c4e8ffe, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:16,739 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A0B199F003B143BC87E60C8C3E83A9CD: , UsingHashComparator=false
2016-05-18 00:13:16,752 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:16,752 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:16,753 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A0B199F003B143BC87E60C8C3E83A9CD: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:16,753 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A0B199F003B143BC87E60C8C3E83A9CD: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:16,754 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] AAB7D6DE35B94AFA9FD9FA3B335E1DA0, for 1 inputs
2016-05-18 00:13:16,755 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:16,755 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:16,756 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BA83B71CC23A4663B2B3DC10D2BC90BC, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:16,757 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F0156A659DDA4644B0229FE289F097BA: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:16,781 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,782 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BA83B71CC23A4663B2B3DC10D2BC90BC, mem on close (mb), free: 52, total: 391, max: 672
2016-05-18 00:13:16,782 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F0156A659DDA4644B0229FE289F097BA
2016-05-18 00:13:16,782 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F0156A659DDA4644B0229FE289F097BA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:16,783 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A0B199F003B143BC87E60C8C3E83A9CD: Starting flush of map output
2016-05-18 00:13:16,783 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A0B199F003B143BC87E60C8C3E83A9CD: Span0.length = 174, perItem = 75
2016-05-18 00:13:16,791 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A0B199F003B143BC87E60C8C3E83A9CD: done sorting span=0, length=174, time=7
2016-05-18 00:13:16,791 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A0B199F003B143BC87E60C8C3E83A9CD: Heap = SpanIterator<0:173> (span=Span[16777216,13104]),
2016-05-18 00:13:16,792 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A0B199F003B143BC87E60C8C3E83A9CD: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_01_000000_0_10030_0/file.out
2016-05-18 00:13:16,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:16 Completed running task attempt: attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:13:16,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530396367_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:13:16,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530396367_0001_1_01_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530396756, Process_Duration=25, Process_End_Time=1463530396781, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=5]]
2016-05-18 00:13:16,804 WARN task.TaskReporter (TaskReporter.java:call(214)) - Exiting TaskReporter thread with pending queue size=2
2016-05-18 00:13:16,806 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530396367_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,806 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,806 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:16,806 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, taskAttemptId=attempt_1463530396367_0001_1_01_000000_0, creationTime=1463530396685, allocationTime=1463530396687, startTime=1463530396696, finishTime=1463530396807, timeTaken=111, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:16,807 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000002
2016-05-18 00:13:16,807 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530396367_0001_1_01_000000_0
2016-05-18 00:13:16,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_FINISHED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, taskId=task_1463530396367_0001_1_01_000000, startTime=1463530396696, finishTime=1463530396807, timeTaken=111, status=SUCCEEDED, successfulAttemptID=attempt_1463530396367_0001_1_01_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396756, Process_Duration=25, Process_End_Time=1463530396781, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=5
2016-05-18 00:13:16,809 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:16,809 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,810 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_FINISHED]: vertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, vertexId=vertex_1463530396367_0001_1_01, initRequestedTime=1463530396555, initedTime=1463530396562, startRequestedTime=1463530396587, startedTime=1463530396588, finishTime=1463530396809, timeTaken=221, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396756, Process_Duration=25, Process_End_Time=1463530396781, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=5, vertexStats=firstTaskStartTime=1463530396696, firstTasksToStart=[ task_1463530396367_0001_1_01_000000 ], lastTaskFinishTime=1463530396807, lastTasksToFinish=[ task_1463530396367_0001_1_01_000000 ], minTaskDuration=111, maxTaskDuration=111, avgTaskDuration=111.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530396367_0001_1_01_000000 ], longestDurationTasks=[ task_1463530396367_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:16,811 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:16,811 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] attempt: attempt_1463530396367_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:16,811 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A0B199F003B143BC87E60C8C3E83A9CD
2016-05-18 00:13:16,811 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, destinationVertexName=A0B199F003B143BC87E60C8C3E83A9CD. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:16,811 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,811 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530396367_0001_00_000002, stoppedTime=1463530396811, exitStatus=0
2016-05-18 00:13:16,812 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:16,812 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:16,812 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_STARTED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, taskId=task_1463530396367_0001_1_02_000000, scheduledTime=1463530396812, launchTime=1463530396812
2016-05-18 00:13:16,813 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,813 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530396367_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:13:16,815 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530396367_0001_00_000003, launchTime=1463530396814
2016-05-18 00:13:16,816 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000003
2016-05-18 00:13:16,817 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530396367_0001_1_02_000000_0] started. Is using containerId: [container_1463530396367_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 00:13:16,817 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0, startTime=1463530396817, containerId=container_1463530396367_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 00:13:16,818 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530396367_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:13:16,818 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:16,818 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530396367_0001_00_000002, Exiting
2016-05-18 00:13:16,818 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530396367_0001_00_000002
2016-05-18 00:13:16,818 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530396367_0001_00_000002
2016-05-18 00:13:16,819 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,819 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530396367_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:16,819 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530396367_0001_00_000002 completed successfully
2016-05-18 00:13:16 Starting to run new task attempt: attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,820 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,820 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,820 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:16,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: A0B199F003B143BC87E60C8C3E83A9CD, VertexParallelism: 1, TaskAttemptID:attempt_1463530396367_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BA83B71CC23A4663B2B3DC10D2BC90BC, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D1DB9DFA72364EF487BCB83C38360FD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:16,821 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:16,821 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,824 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530396367_0001_1_02_000000_0 1 events [0,1) total 1 vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,824 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530396367_0001_1_02_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:16,825 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:16,827 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:16,835 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A0B199F003B143BC87E60C8C3E83A9CD, ordinal: 2
2016-05-18 00:13:16,835 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A0B199F003B143BC87E60C8C3E83A9CD, mem on start (mb), free: 47, total: 391, max: 672
2016-05-18 00:13:16,835 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:16,835 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:16,835 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:16,836 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:16,836 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BA83B71CC23A4663B2B3DC10D2BC90BC:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [D1DB9DFA72364EF487BCB83C38360FD3:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:16,836 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:16,837 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:16,837 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:16,837 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BA83B71CC23A4663B2B3DC10D2BC90BC:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [D1DB9DFA72364EF487BCB83C38360FD3:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:16,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:16,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BA83B71CC23A4663B2B3DC10D2BC90BC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:16,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:16,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:16,839 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:16,840 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BA83B71CC23A4663B2B3DC10D2BC90BC: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:16,841 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BA83B71CC23A4663B2B3DC10D2BC90BC with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:16,842 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BA83B71CC23A4663B2B3DC10D2BC90BC
2016-05-18 00:13:16,842 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:16,842 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,843 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BA83B71CC23A4663B2B3DC10D2BC90BC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:16,848 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530396367_0001_1_01_000000_0_10030} to DISK_DIRECT, csize=7519, dsize=7515, EndTime=1463530396848, TimeTaken=4, Rate=1.79 MB/s
2016-05-18 00:13:16,848 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-18 00:13:16,848 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BA83B71CC23A4663B2B3DC10D2BC90BC
2016-05-18 00:13:16,849 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:16,849 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: D8912D8814E049A48866842D95AD58A1
2016-05-18 00:13:16,849 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BA83B71CC23A4663B2B3DC10D2BC90BC, wasInterrupted=false
2016-05-18 00:13:16,849 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: CB86D93109B744ADBE2FB68F5F7686F5
2016-05-18 00:13:16,850 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:16,850 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] CB86D93109B744ADBE2FB68F5F7686F5
2016-05-18 00:13:16,851 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:16,851 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_01_000000_0_10030/file.out, len=7519, isLocal=true
2016-05-18 00:13:16,851 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 7519 bytes from disk
2016-05-18 00:13:16,851 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:16,852 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:16,852 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D1DB9DFA72364EF487BCB83C38360FD3 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@e53cad3, comparator=cascading.tuple.hadoop.util.TupleComparator@1f28e9fe, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:16,852 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D1DB9DFA72364EF487BCB83C38360FD3: , UsingHashComparator=false
2016-05-18 00:13:16,852 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-18 00:13:16,861 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BA83B71CC23A4663B2B3DC10D2BC90BC
2016-05-18 00:13:16,861 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BA83B71CC23A4663B2B3DC10D2BC90BC: Shuffle Runner thread complete
2016-05-18 00:13:16,873 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:16,874 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:16,874 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D1DB9DFA72364EF487BCB83C38360FD3: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:16,874 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D1DB9DFA72364EF487BCB83C38360FD3: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:16,874 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] D8912D8814E049A48866842D95AD58A1, for 1 inputs
2016-05-18 00:13:16,874 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A0B199F003B143BC87E60C8C3E83A9CD, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:16,875 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BA83B71CC23A4663B2B3DC10D2BC90BC: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:16,897 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,897 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A0B199F003B143BC87E60C8C3E83A9CD, mem on close (mb), free: 149, total: 377, max: 672
2016-05-18 00:13:16,897 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BA83B71CC23A4663B2B3DC10D2BC90BC
2016-05-18 00:13:16,897 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BA83B71CC23A4663B2B3DC10D2BC90BC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:16,898 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D1DB9DFA72364EF487BCB83C38360FD3: Starting flush of map output
2016-05-18 00:13:16,898 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D1DB9DFA72364EF487BCB83C38360FD3: Span0.length = 87, perItem = 81
2016-05-18 00:13:16,900 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D1DB9DFA72364EF487BCB83C38360FD3: done sorting span=0, length=87, time=2
2016-05-18 00:13:16,900 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D1DB9DFA72364EF487BCB83C38360FD3: Heap = SpanIterator<0:86> (span=Span[16777216,7074]),
2016-05-18 00:13:16,901 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D1DB9DFA72364EF487BCB83C38360FD3: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_02_000000_0_10033_0/file.out
2016-05-18 00:13:16,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:16 Completed running task attempt: attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,907 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0, askedToStop=false
2016-05-18 00:13:16,907 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530396367_0001_1_02_000000_0, stopRequested=false
2016-05-18 00:13:16,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530396367_0001_1_02_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530396874, Process_Duration=23, Process_End_Time=1463530396897, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=3]]
2016-05-18 00:13:16,909 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530396367_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,910 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, taskAttemptId=attempt_1463530396367_0001_1_02_000000_0, creationTime=1463530396812, allocationTime=1463530396814, startTime=1463530396817, finishTime=1463530396910, timeTaken=93, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:16,910 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:16,910 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530396367_0001_1_02_000000_0
2016-05-18 00:13:16,911 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_FINISHED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, taskId=task_1463530396367_0001_1_02_000000, startTime=1463530396817, finishTime=1463530396910, timeTaken=93, status=SUCCEEDED, successfulAttemptID=attempt_1463530396367_0001_1_02_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396874, Process_Duration=23, Process_End_Time=1463530396897, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=3
2016-05-18 00:13:16,912 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:16,912 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,912 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,913 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000003
2016-05-18 00:13:16,914 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_FINISHED]: vertexName=A0B199F003B143BC87E60C8C3E83A9CD, vertexId=vertex_1463530396367_0001_1_02, initRequestedTime=1463530396562, initedTime=1463530396569, startRequestedTime=1463530396592, startedTime=1463530396595, finishTime=1463530396912, timeTaken=317, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396874, Process_Duration=23, Process_End_Time=1463530396897, Read_Duration=6, Tuples_Read=174, Tuples_Written=87, Write_Duration=3, vertexStats=firstTaskStartTime=1463530396817, firstTasksToStart=[ task_1463530396367_0001_1_02_000000 ], lastTaskFinishTime=1463530396910, lastTasksToFinish=[ task_1463530396367_0001_1_02_000000 ], minTaskDuration=93, maxTaskDuration=93, avgTaskDuration=93.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530396367_0001_1_02_000000 ], longestDurationTasks=[ task_1463530396367_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:16,914 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:16,915 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530396367_0001_00_000003, stoppedTime=1463530396915, exitStatus=0
2016-05-18 00:13:16,915 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] attempt: attempt_1463530396367_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:16,915 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:16,916 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:16,915 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D1DB9DFA72364EF487BCB83C38360FD3
2016-05-18 00:13:16,916 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A0B199F003B143BC87E60C8C3E83A9CD, destinationVertexName=D1DB9DFA72364EF487BCB83C38360FD3. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:16,916 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530396367_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 00:13:16,916 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,917 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_STARTED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, taskId=task_1463530396367_0001_1_03_000000, scheduledTime=1463530396916, launchTime=1463530396916
2016-05-18 00:13:16,917 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,918 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530396367_0001_00_000004, launchTime=1463530396918
2016-05-18 00:13:16,919 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000004
2016-05-18 00:13:16,919 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530396367_0001_1_03_000000_0] started. Is using containerId: [container_1463530396367_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-18 00:13:16,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0, startTime=1463530396919, containerId=container_1463530396367_0001_00_000004, nodeId=127.0.0.1:0
2016-05-18 00:13:16,921 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16 Starting to run new task attempt: attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,921 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,921 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,921 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:16,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, VertexName: D1DB9DFA72364EF487BCB83C38360FD3, VertexParallelism: 1, TaskAttemptID:attempt_1463530396367_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A0B199F003B143BC87E60C8C3E83A9CD, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=26AF33801AB34C4D8330E7709122DD7D, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:13:16,921 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:16,922 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,923 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530396367_0001_1_03_000000_0 1 events [0,1) total 1 vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,923 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530396367_0001_1_03_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:16,923 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530396367_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 00:13:16,924 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:16,924 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530396367_0001_00_000003, Exiting
2016-05-18 00:13:16,924 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530396367_0001_00_000003
2016-05-18 00:13:16,924 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530396367_0001_00_000003
2016-05-18 00:13:16,925 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530396367_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:16,925 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530396367_0001_00_000003 completed successfully
2016-05-18 00:13:16,926 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:16,927 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:16,936 INFO output.MROutput (MROutput.java:initialize(421)) - 26AF33801AB34C4D8330E7709122DD7D: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:13:16,937 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D1DB9DFA72364EF487BCB83C38360FD3, ordinal: 3
2016-05-18 00:13:16,937 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D1DB9DFA72364EF487BCB83C38360FD3, mem on start (mb), free: 143, total: 377, max: 672
2016-05-18 00:13:16,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:16,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:16,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:16,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:16,938 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[26AF33801AB34C4D8330E7709122DD7D:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A0B199F003B143BC87E60C8C3E83A9CD:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:16,938 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:16,939 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:16,939 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:13:16,939 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[26AF33801AB34C4D8330E7709122DD7D:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A0B199F003B143BC87E60C8C3E83A9CD:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:16,939 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:16,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A0B199F003B143BC87E60C8C3E83A9CD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:16,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:16,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:16,940 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A0B199F003B143BC87E60C8C3E83A9CD: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:16,941 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A0B199F003B143BC87E60C8C3E83A9CD: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:16,942 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A0B199F003B143BC87E60C8C3E83A9CD with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:16,942 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A0B199F003B143BC87E60C8C3E83A9CD
2016-05-18 00:13:16,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:16,944 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,945 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A0B199F003B143BC87E60C8C3E83A9CD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:16,947 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530396367_0001_1_02_000000_0_10033} to DISK_DIRECT, csize=5127, dsize=5123, EndTime=1463530396947, TimeTaken=1, Rate=4.89 MB/s
2016-05-18 00:13:16,947 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A0B199F003B143BC87E60C8C3E83A9CD
2016-05-18 00:13:16,947 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:16,947 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-05-18 00:13:16,947 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A0B199F003B143BC87E60C8C3E83A9CD, wasInterrupted=false
2016-05-18 00:13:16,948 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:16,948 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: CB86D93109B744ADBE2FB68F5F7686F5
2016-05-18 00:13:16,949 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"], id: 26AF33801AB34C4D8330E7709122DD7D
2016-05-18 00:13:16,949 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long"]
2016-05-18 00:13:16,949 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] CB86D93109B744ADBE2FB68F5F7686F5, for 1 inputs
2016-05-18 00:13:16,949 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:16,950 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001_wd/localmode-local-dir/output/attempt_1463530396367_0001_1_02_000000_0_10033/file.out, len=5127, isLocal=true
2016-05-18 00:13:16,950 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 5127 bytes from disk
2016-05-18 00:13:16,950 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:16,950 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:16,951 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-05-18 00:13:16,951 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A0B199F003B143BC87E60C8C3E83A9CD
2016-05-18 00:13:16,951 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A0B199F003B143BC87E60C8C3E83A9CD: Shuffle Runner thread complete
2016-05-18 00:13:16,951 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D1DB9DFA72364EF487BCB83C38360FD3, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:13:16,952 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A0B199F003B143BC87E60C8C3E83A9CD: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:16,963 INFO output.MROutput (MROutput.java:close(526)) - 26AF33801AB34C4D8330E7709122DD7D closed
2016-05-18 00:13:16,964 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463530396367_0001_1_03_000000_0 given a go for committing the task output.
2016-05-18 00:13:16,964 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,964 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D1DB9DFA72364EF487BCB83C38360FD3, mem on close (mb), free: 140, total: 377, max: 672
2016-05-18 00:13:16,965 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A0B199F003B143BC87E60C8C3E83A9CD
2016-05-18 00:13:16,965 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A0B199F003B143BC87E60C8C3E83A9CD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:16,966 INFO output.MROutput (MROutput.java:close(526)) - 26AF33801AB34C4D8330E7709122DD7D closed
2016-05-18 00:13:16,966 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:16 Completed running task attempt: attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,967 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0, askedToStop=false
2016-05-18 00:13:16,967 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530396367_0001_1_03_000000_0, stopRequested=false
2016-05-18 00:13:16,968 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530396367_0001_1_03_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530396952, Process_Duration=12, Process_End_Time=1463530396964, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-18 00:13:16,969 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530396367_0001_1_03_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,969 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:16,970 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:16,970 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, taskAttemptId=attempt_1463530396367_0001_1_03_000000_0, creationTime=1463530396917, allocationTime=1463530396918, startTime=1463530396919, finishTime=1463530396970, timeTaken=51, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:16,970 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530396367_0001_1_03_000000_0
2016-05-18 00:13:16,971 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:TASK_FINISHED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, taskId=task_1463530396367_0001_1_03_000000, startTime=1463530396919, finishTime=1463530396970, timeTaken=51, status=SUCCEEDED, successfulAttemptID=attempt_1463530396367_0001_1_03_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396952, Process_Duration=12, Process_End_Time=1463530396964, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21
2016-05-18 00:13:16,972 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530396367_0001_00_000004
2016-05-18 00:13:16,972 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:16,972 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3]
2016-05-18 00:13:16,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:VERTEX_FINISHED]: vertexName=D1DB9DFA72364EF487BCB83C38360FD3, vertexId=vertex_1463530396367_0001_1_03, initRequestedTime=1463530396570, initedTime=1463530396575, startRequestedTime=1463530396597, startedTime=1463530396598, finishTime=1463530396972, timeTaken=374, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530396952, Process_Duration=12, Process_End_Time=1463530396964, Read_Duration=6, Tuples_Read=87, Tuples_Written=21, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463530396919, firstTasksToStart=[ task_1463530396367_0001_1_03_000000 ], lastTaskFinishTime=1463530396970, lastTasksToFinish=[ task_1463530396367_0001_1_03_000000 ], minTaskDuration=51, maxTaskDuration=51, avgTaskDuration=51.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530396367_0001_1_03_000000 ], longestDurationTasks=[ task_1463530396367_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:16,974 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:16,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530396367_0001_00_000004, stoppedTime=1463530396974, exitStatus=0
2016-05-18 00:13:16,974 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:16,974 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:16,975 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530396367_0001_1_02 [A0B199F003B143BC87E60C8C3E83A9CD]
2016-05-18 00:13:16,975 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530396367_0001_1_01 [BA83B71CC23A4663B2B3DC10D2BC90BC]
2016-05-18 00:13:16,975 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530396367_0001_1_00 [F0156A659DDA4644B0229FE289F097BA]
2016-05-18 00:13:16,975 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463530396367_0001_1
2016-05-18 00:13:16,975 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463530396367_0001_1
2016-05-18 00:13:16,975 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530396367_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:13:16,976 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 26AF33801AB34C4D8330E7709122DD7D for vertex: vertex_1463530396367_0001_1_03 [D1DB9DFA72364EF487BCB83C38360FD3], outputName: 26AF33801AB34C4D8330E7709122DD7D
2016-05-18 00:13:16,976 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530396367_0001_00_000004. Relying on regular task shutdown for it to end
2016-05-18 00:13:16,982 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:26AF33801AB34C4D8330E7709122DD7D of vertex/vertexGroup:D1DB9DFA72364EF487BCB83C38360FD3 isVertexGroupOutput:false
2016-05-18 00:13:16,982 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:16,983 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530396367_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-18 00:13:16,983 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530396367_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:16,983 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530396367_0001_00_000004, Exiting
2016-05-18 00:13:16,983 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530396367_0001_00_000004
2016-05-18 00:13:16,984 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530396367_0001_00_000004
2016-05-18 00:13:16,984 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530396367_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:16,984 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530396367_0001_00_000004 completed successfully
2016-05-18 00:13:16,987 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463530396367_0001_1, queueSize=0
2016-05-18 00:13:16,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530396367_0001_1][Event:DAG_FINISHED]: dagId=dag_1463530396367_0001_1, startTime=1463530396553, finishTime=1463530396982, timeTaken=429, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=870, AM_GC_TIME_MILLIS=19, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=114, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=3, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=49, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=16, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854121587247, Process_Duration=64, Process_End_Time=5854121587311, Read_Duration=15, Tuples_Read=303, Tuples_Written=317, Write_Duration=8, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-18 00:13:16,989 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463530396367_0001_1 finished with state: SUCCEEDED
2016-05-18 00:13:16,989 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530396367_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:13:16 Completed Dag: dag_1463530396367_0001_1
2016-05-18 00:13:16,990 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463530396367_0001_1, dagState=SUCCEEDED
2016-05-18 00:13:16,990 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530396367_0001_1_post
2016-05-18 00:13:16,990 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:16,990 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:13:16,990 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:13:16,991 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, with id=dag_1463530396367_0001_1
2016-05-18 00:13:16,991 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7] euclidean/(1/1) ...anDistance/euclidean/long, with id=dag_1463530396367_0001_1
2016-05-18 00:13:16,997 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...anDistance/euclidean/long, applicationId=application_1463530396367_0001
2016-05-18 00:13:17,002 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:13:17,002 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:13:17,002 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:13:17,002 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:13:17,002 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:13:17,002 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:13:17,003 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:13:17,003 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:13:17,003 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:13:17,003 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:13:17,004 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:13:17,004 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:13:17,004 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:13:17,006 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:13:17,009 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/791790FD005D48ADA40DDDDF5FBE643C/B69E31A7BA02439EADF90B4F5B1551B7/.tez/application_1463530396367_0001
2016-05-18 00:13:17,010 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long/_temporary
2016-05-18 00:13:17,011 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:00.667
2016-05-18 00:13:17,093 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistance/euclidean/long
2016-05-18 00:13:17,128 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:13:17,129 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:13:17,198 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.039
2016-05-18 00:13:17,217 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.057
2016-05-18 00:13:17,218 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:17,218 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 4
2016-05-18 00:13:17,218 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:13:17,239 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt
2016-05-18 00:13:17,258 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short
2016-05-18 00:13:17,408 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting
2016-05-18 00:13:17,408 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:17,409 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-05-18 00:13:17,409 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] parallel execution of steps is enabled: false
2016-05-18 00:13:17,409 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] executing total steps: 1
2016-05-18 00:13:17,410 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] allocating management threads: 1
2016-05-18 00:13:17,411 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] starting step: (1/1) ...anceShort/euclidean/short
2016-05-18 00:13:17,411 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [euclidean] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:13:17,434 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] tez session mode enabled: true
2016-05-18 00:13:17,439 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:17,440 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:13:17,440 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:13:17,449 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001 doesn't exist and is created
2016-05-18 00:13:17,464 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:13:17,464 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:13:17,465 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd
2016-05-18 00:13:17,475 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463530397439_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:13:17,475 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:17,475 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:13:17,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:13:17,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:13:17,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:13:17,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:13:17,478 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:13:17,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:13:17,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:13:17,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:13:17,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:13:17,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:13:17,481 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:13:17,481 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:13:17,481 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:13:17,481 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:17,482 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:13:17,482 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:13:17,482 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:13:17,484 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:13:17,485 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:13:17,485 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:13:17,485 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-log-dir
2016-05-18 00:13:17,486 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-log-dir/history.txt.appattempt_1463530397439_0001_000000, maxErrors=10
2016-05-18 00:13:17,486 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:13:17,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463530397439_0001_000000, appSubmitTime=1463530397474, launchTime=1463530397475
2016-05-18 00:13:17,497 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:58618
2016-05-18 00:13:17,497 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463530397439
2016-05-18 00:13:17,499 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:13:17,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463530397439_0001_000000, startTime=1463530397501
2016-05-18 00:13:17,501 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:13:17,565 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:13:17,565 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:13:17,565 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463530397439_0001, dagName=[39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-18 00:13:17,587 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-18 00:13:17,587 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:13:17,588 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463530397439_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-log-dir/dag_1463530397439_0001_1.dot
2016-05-18 00:13:17,591 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-log-dir/dag_1463530397439_0001_1-tez-dag.pb.txt
2016-05-18 00:13:17,619 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530397439_0001_1
2016-05-18 00:13:17,620 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,620 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-18 00:13:17 Running Dag: dag_1463530397439_0001_1
2016-05-18 00:13:17,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463530397439_0001_1, submitTime=1463530397587
2016-05-18 00:13:17,633 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001/tez-conf.pb" } size: 36897 timestamp: 1463530397000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:17,633 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,635 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:17,636 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001/tez-conf.pb" } size: 36897 timestamp: 1463530397000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:17,637 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:17,638 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001/tez-conf.pb" } size: 36897 timestamp: 1463530397000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:17,638 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:17,643 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001/tez-conf.pb" } size: 36897 timestamp: 1463530397000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:13:17,644 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,644 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:13:17,644 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:13:17,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463530397439_0001_1, initTime=1463530397645
2016-05-18 00:13:17,645 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530397439_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:13:17,646 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:13:17,646 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:DAG_STARTED]: dagID=dag_1463530397439_0001_1, startTime=1463530397646
2016-05-18 00:13:17,646 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530397439_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:13:17,646 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AD5BD13C91FC4CA1B512216EB65C49E6 : {BF19D3B8962A4577BE5072FEFF9A04F9={InputName=BF19D3B8962A4577BE5072FEFF9A04F9}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:13:17,646 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BF19D3B8962A4577BE5072FEFF9A04F9, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:13:17,646 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,646 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,647 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,647 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,648 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:13:17,648 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,648 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BF19D3B8962A4577BE5072FEFF9A04F9 on vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,651 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463530397439_0001, dagName=[39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short
2016-05-18 00:13:17,652 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:17,652 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BF19D3B8962A4577BE5072FEFF9A04F9 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:13:17,652 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,653 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,654 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530397439_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:17,654 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BFE0743528924F94B88DC18601D3F335, vertexId=vertex_1463530397439_0001_1_01, initRequestedTime=1463530397648, initedTime=1463530397654, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:17,654 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:17,655 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,657 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:13:17,657 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 828 Min splitLength: 52428800 New desired splits: 1 Total length: 828 Original splits: 2
2016-05-18 00:13:17,658 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 828 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 828 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:13:17,658 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:13:17,659 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BF19D3B8962A4577BE5072FEFF9A04F9 on vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,660 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:17,660 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,660 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,661 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530397439_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:17,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, vertexId=vertex_1463530397439_0001_1_02, initRequestedTime=1463530397655, initedTime=1463530397661, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:17,662 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:17,662 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,663 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:17,663 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BF19D3B8962A4577BE5072FEFF9A04F9=forAllWorkUnits=true, update=[1]}
2016-05-18 00:13:17,664 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] parallelism set to 1
2016-05-18 00:13:17,664 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:13:17,667 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:13:17,667 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,667 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,668 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530397439_0001_1_03, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:13:17,668 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A], numAdditionalOutputs=1
2016-05-18 00:13:17,669 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=81BA6267A60E47359D72F741451CE540, vertex=vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:13:17,671 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F247AA1E0CE0490AA51BBE85178E228A:81BA6267A60E47359D72F741451CE540 using oldmapred API
2016-05-18 00:13:17,671 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=81BA6267A60E47359D72F741451CE540, vertexName=F247AA1E0CE0490AA51BBE85178E228A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:13:17,673 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, vertexId=vertex_1463530397439_0001_1_03, initRequestedTime=1463530397662, initedTime=1463530397668, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:13:17,673 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:13:17,674 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, vertexId=vertex_1463530397439_0001_1_00, initRequestedTime=1463530397646, initedTime=1463530397673, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:13:17,674 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] with distanceFromRoot: 0
2016-05-18 00:13:17,674 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:13:17,674 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] submitted tez dag to app master: application_1463530397439_0001, with dag id: dag_1463530397439_0001_1
2016-05-18 00:13:17,676 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530397439_0001_1_00, startRequestedTime=1463530397648, startedTime=1463530397675
2016-05-18 00:13:17,676 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463530397439_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:13:17,676 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:17,677 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530397439_0001_1_00 for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] numStartedSources: 1 numSources: 1
2016-05-18 00:13:17,677 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] with distanceFromRoot: 1
2016-05-18 00:13:17,677 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AD5BD13C91FC4CA1B512216EB65C49E6
2016-05-18 00:13:17,677 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530397439_0001_1_01, startRequestedTime=1463530397677, startedTime=1463530397677
2016-05-18 00:13:17,678 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:17,678 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530397439_0001_1_01 for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] numStartedSources: 1 numSources: 1
2016-05-18 00:13:17,678 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] with distanceFromRoot: 2
2016-05-18 00:13:17,678 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530397439_0001_1_02, startRequestedTime=1463530397678, startedTime=1463530397679
2016-05-18 00:13:17,681 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:17,681 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_STARTED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, taskId=task_1463530397439_0001_1_00_000000, scheduledTime=1463530397681, launchTime=1463530397681
2016-05-18 00:13:17,682 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463530397439_0001_1_02 for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] numStartedSources: 1 numSources: 1
2016-05-18 00:13:17,684 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] with distanceFromRoot: 3
2016-05-18 00:13:17,684 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,684 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463530397439_0001_1_03, startRequestedTime=1463530397682, startedTime=1463530397684
2016-05-18 00:13:17,685 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:13:17,684 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: ADB3FFC7562442BD8D61B6A52A118784 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:17,686 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F247AA1E0CE0490AA51BBE85178E228A with 0 source tasks and 1 pending tasks
2016-05-18 00:13:17,688 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:13:17,689 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BFE0743528924F94B88DC18601D3F335 in vertex: ADB3FFC7562442BD8D61B6A52A118784 numBipartiteSourceTasks: 1
2016-05-18 00:13:17,688 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BFE0743528924F94B88DC18601D3F335 with 0 source tasks and 1 pending tasks
2016-05-18 00:13:17,689 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: ADB3FFC7562442BD8D61B6A52A118784 in vertex: F247AA1E0CE0490AA51BBE85178E228A numBipartiteSourceTasks: 1
2016-05-18 00:13:17,690 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AD5BD13C91FC4CA1B512216EB65C49E6 in vertex: BFE0743528924F94B88DC18601D3F335 numBipartiteSourceTasks: 1
2016-05-18 00:13:17,691 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530397439_0001_00_000001, launchTime=1463530397691
2016-05-18 00:13:17,692 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000001
2016-05-18 00:13:17,693 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530397439_0001_1_00_000000_0] started. Is using containerId: [container_1463530397439_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:13:17,693 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0, startTime=1463530397692, containerId=container_1463530397439_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:13:17,694 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17 Starting to run new task attempt: attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,694 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,694 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,694 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:17,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: AD5BD13C91FC4CA1B512216EB65C49E6, VertexParallelism: 1, TaskAttemptID:attempt_1463530397439_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BF19D3B8962A4577BE5072FEFF9A04F9, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BFE0743528924F94B88DC18601D3F335, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:17,695 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:17,697 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,697 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463530397439_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:13:17,697 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530397439_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:13:17,702 INFO input.MRInput (MRInput.java:initialize(440)) - BF19D3B8962A4577BE5072FEFF9A04F9 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:13:17,703 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BF19D3B8962A4577BE5072FEFF9A04F9
2016-05-18 00:13:17,703 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:17,712 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AD5BD13C91FC4CA1B512216EB65C49E6, ordinal: 0
2016-05-18 00:13:17,712 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AD5BD13C91FC4CA1B512216EB65C49E6, mem on start (mb), free: 216, total: 390, max: 672
2016-05-18 00:13:17,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:17,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:17,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:17,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:17,713 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BF19D3B8962A4577BE5072FEFF9A04F9:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BFE0743528924F94B88DC18601D3F335:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:17,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:17,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:17,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:13:17,714 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BF19D3B8962A4577BE5072FEFF9A04F9:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BFE0743528924F94B88DC18601D3F335:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:17,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:17,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BF19D3B8962A4577BE5072FEFF9A04F9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:17,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:17,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:17,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BF19D3B8962A4577BE5072FEFF9A04F9
2016-05-18 00:13:17,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:17,716 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,718 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:17,719 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"] streamed: true, id: BF19D3B8962A4577BE5072FEFF9A04F9
2016-05-18 00:13:17,719 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]], id: 0BFC294608C2457C85C16DE19A5D9D33
2016-05-18 00:13:17,719 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0BFC294608C2457C85C16DE19A5D9D33
2016-05-18 00:13:17,720 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:0+414, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt:414+414], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=828}
2016-05-18 00:13:17,720 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BF19D3B8962A4577BE5072FEFF9A04F9 initialized RecordReader from event
2016-05-18 00:13:17,722 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BFE0743528924F94B88DC18601D3F335 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@71a0990e, comparator=cascading.tuple.hadoop.util.TupleComparator@66a0a069, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:17,722 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BFE0743528924F94B88DC18601D3F335: , UsingHashComparator=false
2016-05-18 00:13:17,735 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:17,735 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:17,735 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BFE0743528924F94B88DC18601D3F335: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:17,735 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BFE0743528924F94B88DC18601D3F335: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:17,736 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/critics.txt"]
2016-05-18 00:13:17,736 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AD5BD13C91FC4CA1B512216EB65C49E6, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:17,740 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,740 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AD5BD13C91FC4CA1B512216EB65C49E6, mem on close (mb), free: 115, total: 390, max: 672
2016-05-18 00:13:17,740 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BFE0743528924F94B88DC18601D3F335: Starting flush of map output
2016-05-18 00:13:17,740 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BFE0743528924F94B88DC18601D3F335: Span0.length = 35, perItem = 47
2016-05-18 00:13:17,741 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BFE0743528924F94B88DC18601D3F335: done sorting span=0, length=35, time=1
2016-05-18 00:13:17,741 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BFE0743528924F94B88DC18601D3F335: Heap = SpanIterator<0:34> (span=Span[16777216,1676]),
2016-05-18 00:13:17,743 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BFE0743528924F94B88DC18601D3F335: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_00_000000_0_10038_0/file.out
2016-05-18 00:13:17,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:17 Completed running task attempt: attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,748 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:13:17,748 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530397439_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:13:17,749 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530397439_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463530397736, Process_Duration=4, Process_End_Time=1463530397740, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=7]]
2016-05-18 00:13:17,750 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530397439_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,750 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,750 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:17,750 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,751 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000001
2016-05-18 00:13:17,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, taskAttemptId=attempt_1463530397439_0001_1_00_000000_0, creationTime=1463530397682, allocationTime=1463530397691, startTime=1463530397692, finishTime=1463530397750, timeTaken=58, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:17,751 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530397439_0001_1_00_000000_0
2016-05-18 00:13:17,753 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_FINISHED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, taskId=task_1463530397439_0001_1_00_000000, startTime=1463530397692, finishTime=1463530397751, timeTaken=59, status=SUCCEEDED, successfulAttemptID=attempt_1463530397439_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530397736, Process_Duration=4, Process_End_Time=1463530397740, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7
2016-05-18 00:13:17,753 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:17,753 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:17,754 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_FINISHED]: vertexName=AD5BD13C91FC4CA1B512216EB65C49E6, vertexId=vertex_1463530397439_0001_1_00, initRequestedTime=1463530397646, initedTime=1463530397673, startRequestedTime=1463530397648, startedTime=1463530397675, finishTime=1463530397753, timeTaken=78, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=35, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=35, OUTPUT_BYTES=1676, OUTPUT_BYTES_WITH_OVERHEAD=1085, OUTPUT_BYTES_PHYSICAL=1089, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463530397736, Process_Duration=4, Process_End_Time=1463530397740, Read_Duration=2, Tuples_Read=7, Tuples_Written=35, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=7, vertexStats=firstTaskStartTime=1463530397692, firstTasksToStart=[ task_1463530397439_0001_1_00_000000 ], lastTaskFinishTime=1463530397750, lastTasksToFinish=[ task_1463530397439_0001_1_00_000000 ], minTaskDuration=58, maxTaskDuration=58, avgTaskDuration=58.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530397439_0001_1_00_000000 ], longestDurationTasks=[ task_1463530397439_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:17,755 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:17,755 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530397439_0001_00_000001, stoppedTime=1463530397755, exitStatus=0
2016-05-18 00:13:17,755 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] attempt: attempt_1463530397439_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:17,756 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:17,756 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BFE0743528924F94B88DC18601D3F335
2016-05-18 00:13:17,756 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:17,756 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AD5BD13C91FC4CA1B512216EB65C49E6, destinationVertexName=BFE0743528924F94B88DC18601D3F335. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:17,756 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530397439_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:13:17,756 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,757 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_STARTED]: vertexName=BFE0743528924F94B88DC18601D3F335, taskId=task_1463530397439_0001_1_01_000000, scheduledTime=1463530397757, launchTime=1463530397757
2016-05-18 00:13:17,757 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530397439_0001_00_000002, launchTime=1463530397760
2016-05-18 00:13:17,761 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000002
2016-05-18 00:13:17,761 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530397439_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:13:17,762 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:17,762 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,762 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530397439_0001_00_000001, Exiting
2016-05-18 00:13:17,763 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530397439_0001_1_01_000000_0] started. Is using containerId: [container_1463530397439_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:13:17 Starting to run new task attempt: attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,763 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,763 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,763 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530397439_0001_00_000001
2016-05-18 00:13:17,763 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:17,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: BFE0743528924F94B88DC18601D3F335, VertexParallelism: 1, TaskAttemptID:attempt_1463530397439_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AD5BD13C91FC4CA1B512216EB65C49E6, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ADB3FFC7562442BD8D61B6A52A118784, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:17,764 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:17,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BFE0743528924F94B88DC18601D3F335, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0, startTime=1463530397762, containerId=container_1463530397439_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:13:17,763 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530397439_0001_00_000001
2016-05-18 00:13:17,768 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,768 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530397439_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,768 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530397439_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:17,769 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530397439_0001_00_000001 completed successfully
2016-05-18 00:13:17,768 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530397439_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:17,774 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:17,774 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:17,785 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BFE0743528924F94B88DC18601D3F335, ordinal: 1
2016-05-18 00:13:17,785 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BFE0743528924F94B88DC18601D3F335, mem on start (mb), free: 110, total: 390, max: 672
2016-05-18 00:13:17,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:17,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:17,786 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:17,786 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:17,786 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[ADB3FFC7562442BD8D61B6A52A118784:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [AD5BD13C91FC4CA1B512216EB65C49E6:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:17,786 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:17,788 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:17,788 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:17,788 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[ADB3FFC7562442BD8D61B6A52A118784:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [AD5BD13C91FC4CA1B512216EB65C49E6:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:17,789 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:17,789 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AD5BD13C91FC4CA1B512216EB65C49E6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:17,791 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AD5BD13C91FC4CA1B512216EB65C49E6: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:17,792 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AD5BD13C91FC4CA1B512216EB65C49E6: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:17,794 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AD5BD13C91FC4CA1B512216EB65C49E6 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:17,794 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:17,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:17,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AD5BD13C91FC4CA1B512216EB65C49E6
2016-05-18 00:13:17,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:17,798 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,798 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AD5BD13C91FC4CA1B512216EB65C49E6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:17,800 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530397439_0001_1_00_000000_0_10038} to DISK_DIRECT, csize=1089, dsize=1085, EndTime=1463530397800, TimeTaken=1, Rate=1.04 MB/s
2016-05-18 00:13:17,800 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AD5BD13C91FC4CA1B512216EB65C49E6
2016-05-18 00:13:17,801 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:17,801 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AD5BD13C91FC4CA1B512216EB65C49E6, wasInterrupted=false
2016-05-18 00:13:17,802 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:17,803 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]]
2016-05-18 00:13:17,804 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:17,804 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - AD5BD13C91FC4CA1B512216EB65C49E6: Already shutdown. Ignoring fetch complete
2016-05-18 00:13:17,805 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_00_000000_0_10038/file.out, len=1089, isLocal=true
2016-05-18 00:13:17,805 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 1089 bytes from disk
2016-05-18 00:13:17,805 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:17,805 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:17,805 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] streamed: true, id: 0BFC294608C2457C85C16DE19A5D9D33
2016-05-18 00:13:17,805 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{?}:ALL]], id: C097999CF0824BD5833A28E0299A741E
2016-05-18 00:13:17,805 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 1061 bytes
2016-05-18 00:13:17,805 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C097999CF0824BD5833A28E0299A741E
2016-05-18 00:13:17,806 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AD5BD13C91FC4CA1B512216EB65C49E6
2016-05-18 00:13:17,806 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AD5BD13C91FC4CA1B512216EB65C49E6: Shuffle Runner thread complete
2016-05-18 00:13:17,809 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - ADB3FFC7562442BD8D61B6A52A118784 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@645eca7e, comparator=cascading.tuple.hadoop.util.TupleComparator@104f6bbc, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:17,810 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for ADB3FFC7562442BD8D61B6A52A118784: , UsingHashComparator=false
2016-05-18 00:13:17,832 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:17,832 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:17,833 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for ADB3FFC7562442BD8D61B6A52A118784: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:17,833 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - ADB3FFC7562442BD8D61B6A52A118784: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:17,834 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(euclidean)[by:[{1}:'movie'][numSelfJoins:1]] 0BFC294608C2457C85C16DE19A5D9D33, for 1 inputs
2016-05-18 00:13:17,834 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:17,835 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:13:17,835 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BFE0743528924F94B88DC18601D3F335, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:17,836 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AD5BD13C91FC4CA1B512216EB65C49E6: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:17,860 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,860 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BFE0743528924F94B88DC18601D3F335, mem on close (mb), free: 157, total: 390, max: 672
2016-05-18 00:13:17,861 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AD5BD13C91FC4CA1B512216EB65C49E6
2016-05-18 00:13:17,861 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AD5BD13C91FC4CA1B512216EB65C49E6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:17,861 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - ADB3FFC7562442BD8D61B6A52A118784: Starting flush of map output
2016-05-18 00:13:17,861 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - ADB3FFC7562442BD8D61B6A52A118784: Span0.length = 174, perItem = 75
2016-05-18 00:13:17,866 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - ADB3FFC7562442BD8D61B6A52A118784: done sorting span=0, length=174, time=3
2016-05-18 00:13:17,866 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - ADB3FFC7562442BD8D61B6A52A118784: Heap = SpanIterator<0:173> (span=Span[16777216,13104]),
2016-05-18 00:13:17,867 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - ADB3FFC7562442BD8D61B6A52A118784: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_01_000000_0_10042_0/file.out
2016-05-18 00:13:17,873 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:17 Completed running task attempt: attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,874 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:13:17,875 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530397439_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:13:17,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530397439_0001_1_01_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530397835, Process_Duration=25, Process_End_Time=1463530397860, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=8]]
2016-05-18 00:13:17,877 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530397439_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,877 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,878 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BFE0743528924F94B88DC18601D3F335, taskAttemptId=attempt_1463530397439_0001_1_01_000000_0, creationTime=1463530397757, allocationTime=1463530397759, startTime=1463530397762, finishTime=1463530397877, timeTaken=115, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:17,878 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:17,878 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,878 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000002
2016-05-18 00:13:17,878 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530397439_0001_1_01_000000_0
2016-05-18 00:13:17,880 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_FINISHED]: vertexName=BFE0743528924F94B88DC18601D3F335, taskId=task_1463530397439_0001_1_01_000000, startTime=1463530397762, finishTime=1463530397879, timeTaken=117, status=SUCCEEDED, successfulAttemptID=attempt_1463530397439_0001_1_01_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530397835, Process_Duration=25, Process_End_Time=1463530397860, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=8
2016-05-18 00:13:17,880 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:17,880 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:17,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_FINISHED]: vertexName=BFE0743528924F94B88DC18601D3F335, vertexId=vertex_1463530397439_0001_1_01, initRequestedTime=1463530397648, initedTime=1463530397654, startRequestedTime=1463530397677, startedTime=1463530397677, finishTime=1463530397880, timeTaken=203, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=6, REDUCE_INPUT_RECORDS=35, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=209, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=174, OUTPUT_BYTES=13104, OUTPUT_BYTES_WITH_OVERHEAD=7515, OUTPUT_BYTES_PHYSICAL=7519, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=1089, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=1089, SHUFFLE_BYTES_DECOMPRESSED=1085, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=1089, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530397835, Process_Duration=25, Process_End_Time=1463530397860, Read_Duration=1, Tuples_Read=35, Tuples_Written=174, Write_Duration=8, vertexStats=firstTaskStartTime=1463530397762, firstTasksToStart=[ task_1463530397439_0001_1_01_000000 ], lastTaskFinishTime=1463530397877, lastTasksToFinish=[ task_1463530397439_0001_1_01_000000 ], minTaskDuration=115, maxTaskDuration=115, avgTaskDuration=115.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530397439_0001_1_01_000000 ], longestDurationTasks=[ task_1463530397439_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:17,882 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:17,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530397439_0001_00_000002, stoppedTime=1463530397882, exitStatus=0
2016-05-18 00:13:17,883 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] attempt: attempt_1463530397439_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:17,883 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:17,883 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:17,883 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ADB3FFC7562442BD8D61B6A52A118784
2016-05-18 00:13:17,884 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530397439_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:13:17,884 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BFE0743528924F94B88DC18601D3F335, destinationVertexName=ADB3FFC7562442BD8D61B6A52A118784. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:17,884 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,885 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_STARTED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, taskId=task_1463530397439_0001_1_02_000000, scheduledTime=1463530397884, launchTime=1463530397884
2016-05-18 00:13:17,885 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530397439_0001_00_000003, launchTime=1463530397888
2016-05-18 00:13:17,889 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000003
2016-05-18 00:13:17,889 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530397439_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:13:17,890 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:17,890 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530397439_0001_00_000002, Exiting
2016-05-18 00:13:17,890 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530397439_0001_00_000002
2016-05-18 00:13:17,890 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,891 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530397439_0001_1_02_000000_0] started. Is using containerId: [container_1463530397439_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 00:13:17,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0, startTime=1463530397890, containerId=container_1463530397439_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 00:13:17,890 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530397439_0001_00_000002
2016-05-18 00:13:17,892 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530397439_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:17,892 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530397439_0001_00_000002 completed successfully
2016-05-18 00:13:17 Starting to run new task attempt: attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,894 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,894 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,895 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:17,895 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: ADB3FFC7562442BD8D61B6A52A118784, VertexParallelism: 1, TaskAttemptID:attempt_1463530397439_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BFE0743528924F94B88DC18601D3F335, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=F247AA1E0CE0490AA51BBE85178E228A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:13:17,896 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:17,896 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,897 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530397439_0001_1_02_000000_0 1 events [0,1) total 1 vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,897 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530397439_0001_1_02_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:17,899 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:17,900 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:17,907 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ADB3FFC7562442BD8D61B6A52A118784, ordinal: 2
2016-05-18 00:13:17,907 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ADB3FFC7562442BD8D61B6A52A118784, mem on start (mb), free: 150, total: 390, max: 672
2016-05-18 00:13:17,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:17,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:17,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:17,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:17,908 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BFE0743528924F94B88DC18601D3F335:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [F247AA1E0CE0490AA51BBE85178E228A:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:13:17,908 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:17,908 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:17,909 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 24, TotalRequested: 316250512, TotalRequestedScaled: 1.58125256E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.45
2016-05-18 00:13:17,909 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BFE0743528924F94B88DC18601D3F335:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [F247AA1E0CE0490AA51BBE85178E228A:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:13:17,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:17,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BFE0743528924F94B88DC18601D3F335 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:17,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:17,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:17,910 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BFE0743528924F94B88DC18601D3F335: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:17,910 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BFE0743528924F94B88DC18601D3F335: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:17,911 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BFE0743528924F94B88DC18601D3F335 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:17,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BFE0743528924F94B88DC18601D3F335
2016-05-18 00:13:17,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:17,913 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,913 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BFE0743528924F94B88DC18601D3F335: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:17,917 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{?}:ALL]]
2016-05-18 00:13:17,917 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530397439_0001_1_01_000000_0_10042} to DISK_DIRECT, csize=7519, dsize=7515, EndTime=1463530397917, TimeTaken=3, Rate=2.39 MB/s
2016-05-18 00:13:17,917 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BFE0743528924F94B88DC18601D3F335
2016-05-18 00:13:17,917 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:17,918 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BFE0743528924F94B88DC18601D3F335, wasInterrupted=false
2016-05-18 00:13:17,918 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{?}:ALL]] streamed: true, id: C097999CF0824BD5833A28E0299A741E
2016-05-18 00:13:17,919 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(euclidean)[by:[{2}:'name1', 'name2']], id: 1E30B1937F9C405EAC6918433638BC41
2016-05-18 00:13:17,919 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 1E30B1937F9C405EAC6918433638BC41
2016-05-18 00:13:17,918 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 00:13:17,921 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:17,921 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F247AA1E0CE0490AA51BBE85178E228A using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1945c001, comparator=cascading.tuple.hadoop.util.TupleComparator@70bde632, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:13:17,922 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_01_000000_0_10042/file.out, len=7519, isLocal=true
2016-05-18 00:13:17,922 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 7519 bytes from disk
2016-05-18 00:13:17,922 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:17,922 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:17,922 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F247AA1E0CE0490AA51BBE85178E228A: , UsingHashComparator=false
2016-05-18 00:13:17,922 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 7443 bytes
2016-05-18 00:13:17,923 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BFE0743528924F94B88DC18601D3F335
2016-05-18 00:13:17,923 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BFE0743528924F94B88DC18601D3F335: Shuffle Runner thread complete
2016-05-18 00:13:17,935 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:13:17,935 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:13:17,936 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F247AA1E0CE0490AA51BBE85178E228A: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:13:17,936 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F247AA1E0CE0490AA51BBE85178E228A: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:13:17,936 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{?}:ALL]] C097999CF0824BD5833A28E0299A741E, for 1 inputs
2016-05-18 00:13:17,937 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ADB3FFC7562442BD8D61B6A52A118784, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:13:17,937 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BFE0743528924F94B88DC18601D3F335: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:17,966 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,966 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ADB3FFC7562442BD8D61B6A52A118784, mem on close (mb), free: 47, total: 390, max: 672
2016-05-18 00:13:17,967 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BFE0743528924F94B88DC18601D3F335
2016-05-18 00:13:17,967 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BFE0743528924F94B88DC18601D3F335: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:17,967 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F247AA1E0CE0490AA51BBE85178E228A: Starting flush of map output
2016-05-18 00:13:17,968 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F247AA1E0CE0490AA51BBE85178E228A: Span0.length = 87, perItem = 81
2016-05-18 00:13:17,969 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F247AA1E0CE0490AA51BBE85178E228A: done sorting span=0, length=87, time=1
2016-05-18 00:13:17,970 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F247AA1E0CE0490AA51BBE85178E228A: Heap = SpanIterator<0:86> (span=Span[16777216,7074]),
2016-05-18 00:13:17,971 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F247AA1E0CE0490AA51BBE85178E228A: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_02_000000_0_10045_0/file.out
2016-05-18 00:13:17,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:17 Completed running task attempt: attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0, askedToStop=false
2016-05-18 00:13:17,977 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530397439_0001_1_02_000000_0, stopRequested=false
2016-05-18 00:13:17,977 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530397439_0001_1_02_000000_0: Counters: 40 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530397937, Process_Duration=29, Process_End_Time=1463530397966, Read_Duration=18, Tuples_Read=174, Tuples_Written=87, Write_Duration=1]]
2016-05-18 00:13:17,979 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530397439_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,979 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, taskAttemptId=attempt_1463530397439_0001_1_02_000000_0, creationTime=1463530397885, allocationTime=1463530397887, startTime=1463530397890, finishTime=1463530397979, timeTaken=89, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:17,979 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530397439_0001_1_02_000000_0
2016-05-18 00:13:17,980 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_FINISHED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, taskId=task_1463530397439_0001_1_02_000000, startTime=1463530397890, finishTime=1463530397979, timeTaken=89, status=SUCCEEDED, successfulAttemptID=attempt_1463530397439_0001_1_02_000000_0, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530397937, Process_Duration=29, Process_End_Time=1463530397966, Read_Duration=18, Tuples_Read=174, Tuples_Written=87, Write_Duration=1
2016-05-18 00:13:17,979 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:17,981 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:17,981 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:17,981 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:17,981 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,982 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000003
2016-05-18 00:13:17,982 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_FINISHED]: vertexName=ADB3FFC7562442BD8D61B6A52A118784, vertexId=vertex_1463530397439_0001_1_02, initRequestedTime=1463530397655, initedTime=1463530397661, startRequestedTime=1463530397678, startedTime=1463530397679, finishTime=1463530397981, timeTaken=302, status=SUCCEEDED, diagnostics=, counters=Counters: 40, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=87, REDUCE_INPUT_RECORDS=174, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=261, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=87, OUTPUT_BYTES=7074, OUTPUT_BYTES_WITH_OVERHEAD=5123, OUTPUT_BYTES_PHYSICAL=5127, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=7519, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=7519, SHUFFLE_BYTES_DECOMPRESSED=7515, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=7519, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530397937, Process_Duration=29, Process_End_Time=1463530397966, Read_Duration=18, Tuples_Read=174, Tuples_Written=87, Write_Duration=1, vertexStats=firstTaskStartTime=1463530397890, firstTasksToStart=[ task_1463530397439_0001_1_02_000000 ], lastTaskFinishTime=1463530397979, lastTasksToFinish=[ task_1463530397439_0001_1_02_000000 ], minTaskDuration=89, maxTaskDuration=89, avgTaskDuration=89.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530397439_0001_1_02_000000 ], longestDurationTasks=[ task_1463530397439_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:17,982 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:17,983 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530397439_0001_00_000003, stoppedTime=1463530397983, exitStatus=0
2016-05-18 00:13:17,983 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] attempt: attempt_1463530397439_0001_1_02_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:13:17,983 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F247AA1E0CE0490AA51BBE85178E228A
2016-05-18 00:13:17,983 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=ADB3FFC7562442BD8D61B6A52A118784, destinationVertexName=F247AA1E0CE0490AA51BBE85178E228A. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:13:17,983 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,984 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:17,984 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:17,984 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530397439_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 00:13:17,984 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_STARTED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, taskId=task_1463530397439_0001_1_03_000000, scheduledTime=1463530397984, launchTime=1463530397984
2016-05-18 00:13:17,985 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463530397439_0001_00_000004, launchTime=1463530397987
2016-05-18 00:13:17,987 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000004
2016-05-18 00:13:17,988 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463530397439_0001_1_03_000000_0] started. Is using containerId: [container_1463530397439_0001_00_000004] on NM: [127.0.0.1:0]
2016-05-18 00:13:17,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0, startTime=1463530397988, containerId=container_1463530397439_0001_00_000004, nodeId=127.0.0.1:0
2016-05-18 00:13:17,989 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000004: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:17 Starting to run new task attempt: attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:17,989 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:17,989 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:17,989 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:13:17,990 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, VertexName: F247AA1E0CE0490AA51BBE85178E228A, VertexParallelism: 1, TaskAttemptID:attempt_1463530397439_0001_1_03_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ADB3FFC7562442BD8D61B6A52A118784, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=81BA6267A60E47359D72F741451CE540, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:13:17,990 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:13:17,990 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:17,992 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530397439_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 00:13:17,993 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:17,993 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:13:17,993 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530397439_0001_00_000003, Exiting
2016-05-18 00:13:17,993 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530397439_0001_00_000003
2016-05-18 00:13:17,993 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530397439_0001_00_000003
2016-05-18 00:13:17,994 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530397439_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:17,994 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530397439_0001_00_000003 completed successfully
2016-05-18 00:13:17,995 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:13:17,997 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463530397439_0001_1_03_000000_0 1 events [0,1) total 1 vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:17,998 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463530397439_0001_1_03_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:13:18,003 INFO output.MROutput (MROutput.java:initialize(421)) - 81BA6267A60E47359D72F741451CE540: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:13:18,006 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F247AA1E0CE0490AA51BBE85178E228A, ordinal: 3
2016-05-18 00:13:18,006 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F247AA1E0CE0490AA51BBE85178E228A, mem on start (mb), free: 42, total: 390, max: 672
2016-05-18 00:13:18,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:13:18,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:13:18,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:13:18,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:13:18,006 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[81BA6267A60E47359D72F741451CE540:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [ADB3FFC7562442BD8D61B6A52A118784:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:13:18,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:13:18,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:13:18,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:13:18,007 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[81BA6267A60E47359D72F741451CE540:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [ADB3FFC7562442BD8D61B6A52A118784:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:13:18,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:13:18,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: ADB3FFC7562442BD8D61B6A52A118784 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:13:18,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:13:18,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:13:18,008 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - ADB3FFC7562442BD8D61B6A52A118784: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:13:18,009 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - ADB3FFC7562442BD8D61B6A52A118784: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:13:18,009 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: ADB3FFC7562442BD8D61B6A52A118784 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:13:18,011 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: ADB3FFC7562442BD8D61B6A52A118784
2016-05-18 00:13:18,011 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:13:18,012 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:18,013 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ADB3FFC7562442BD8D61B6A52A118784: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:13:18,015 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463530397439_0001_1_02_000000_0_10045} to DISK_DIRECT, csize=5127, dsize=5123, EndTime=1463530398015, TimeTaken=1, Rate=4.89 MB/s
2016-05-18 00:13:18,016 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : ADB3FFC7562442BD8D61B6A52A118784
2016-05-18 00:13:18,016 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(euclidean)[by:[{2}:'name1', 'name2']]
2016-05-18 00:13:18,016 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:18,017 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] streamed: true, id: 1E30B1937F9C405EAC6918433638BC41
2016-05-18 00:13:18,017 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: ADB3FFC7562442BD8D61B6A52A118784, wasInterrupted=false
2016-05-18 00:13:18,017 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:13:18,018 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"], id: 81BA6267A60E47359D72F741451CE540
2016-05-18 00:13:18,018 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - ADB3FFC7562442BD8D61B6A52A118784: Already shutdown. Ignoring fetch complete
2016-05-18 00:13:18,018 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:13:18,200 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short"]
2016-05-18 00:13:18,200 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(euclidean)[by:[{2}:'name1', 'name2']] 1E30B1937F9C405EAC6918433638BC41, for 1 inputs
2016-05-18 00:13:18,200 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001_wd/localmode-local-dir/output/attempt_1463530397439_0001_1_02_000000_0_10045/file.out, len=5127, isLocal=true
2016-05-18 00:13:18,201 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 5127 bytes from disk
2016-05-18 00:13:18,201 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:13:18,201 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:13:18,202 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 5084 bytes
2016-05-18 00:13:18,202 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : ADB3FFC7562442BD8D61B6A52A118784
2016-05-18 00:13:18,202 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - ADB3FFC7562442BD8D61B6A52A118784: Shuffle Runner thread complete
2016-05-18 00:13:18,203 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F247AA1E0CE0490AA51BBE85178E228A, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:13:18,203 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - ADB3FFC7562442BD8D61B6A52A118784: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:13:18,218 INFO output.MROutput (MROutput.java:close(526)) - 81BA6267A60E47359D72F741451CE540 closed
2016-05-18 00:13:18,219 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463530397439_0001_1_03_000000_0 given a go for committing the task output.
2016-05-18 00:13:18,220 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:18,220 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F247AA1E0CE0490AA51BBE85178E228A, mem on close (mb), free: 341, total: 361, max: 672
2016-05-18 00:13:18,220 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: ADB3FFC7562442BD8D61B6A52A118784
2016-05-18 00:13:18,220 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ADB3FFC7562442BD8D61B6A52A118784: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:13:18,221 INFO output.MROutput (MROutput.java:close(526)) - 81BA6267A60E47359D72F741451CE540 closed
2016-05-18 00:13:18,221 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:13:18 Completed running task attempt: attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:18,222 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0, askedToStop=false
2016-05-18 00:13:18,223 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463530397439_0001_1_03_000000_0, stopRequested=false
2016-05-18 00:13:18,223 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463530397439_0001_1_03_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=194, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463530398203, Process_Duration=17, Process_End_Time=1463530398220, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=21]]
2016-05-18 00:13:18,225 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463530397439_0001_1_03_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:18,225 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, taskAttemptId=attempt_1463530397439_0001_1_03_000000_0, creationTime=1463530397985, allocationTime=1463530397987, startTime=1463530397988, finishTime=1463530398225, timeTaken=237, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:13:18,225 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:13:18,225 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:13:18,225 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463530397439_0001_1_03_000000_0
2016-05-18 00:13:18,226 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:18,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:TASK_FINISHED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, taskId=task_1463530397439_0001_1_03_000000, startTime=1463530397988, finishTime=1463530398225, timeTaken=237, status=SUCCEEDED, successfulAttemptID=attempt_1463530397439_0001_1_03_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=194, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530398203, Process_Duration=17, Process_End_Time=1463530398220, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21
2016-05-18 00:13:18,226 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:13:18,227 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A]
2016-05-18 00:13:18,226 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463530397439_0001_00_000004
2016-05-18 00:13:18,228 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:VERTEX_FINISHED]: vertexName=F247AA1E0CE0490AA51BBE85178E228A, vertexId=vertex_1463530397439_0001_1_03, initRequestedTime=1463530397662, initedTime=1463530397668, startRequestedTime=1463530397682, startedTime=1463530397684, finishTime=1463530398227, timeTaken=543, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=21, REDUCE_INPUT_RECORDS=87, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=87, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=21, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=5127, SHUFFLE_BYTES=5127, SHUFFLE_BYTES_DECOMPRESSED=5123, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=5127, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=194, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463530398203, Process_Duration=17, Process_End_Time=1463530398220, Read_Duration=3, Tuples_Read=87, Tuples_Written=21, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=21, vertexStats=firstTaskStartTime=1463530397988, firstTasksToStart=[ task_1463530397439_0001_1_03_000000 ], lastTaskFinishTime=1463530398225, lastTasksToFinish=[ task_1463530397439_0001_1_03_000000 ], minTaskDuration=237, maxTaskDuration=237, avgTaskDuration=237.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463530397439_0001_1_03_000000 ], longestDurationTasks=[ task_1463530397439_0001_1_03_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:13:18,228 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:13:18,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463530397439_0001_00_000004, stoppedTime=1463530398229, exitStatus=0
2016-05-18 00:13:18,229 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A] completed., numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4
2016-05-18 00:13:18,229 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:18,229 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530397439_0001_1_00 [AD5BD13C91FC4CA1B512216EB65C49E6]
2016-05-18 00:13:18,230 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530397439_0001_1_01 [BFE0743528924F94B88DC18601D3F335]
2016-05-18 00:13:18,230 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463530397439_0001_1_02 [ADB3FFC7562442BD8D61B6A52A118784]
2016-05-18 00:13:18,230 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463530397439_0001_1
2016-05-18 00:13:18,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463530397439_0001_1
2016-05-18 00:13:18,231 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 81BA6267A60E47359D72F741451CE540 for vertex: vertex_1463530397439_0001_1_03 [F247AA1E0CE0490AA51BBE85178E228A], outputName: 81BA6267A60E47359D72F741451CE540
2016-05-18 00:13:18,235 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530397439_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:13:18,236 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463530397439_0001_00_000004. Relying on regular task shutdown for it to end
2016-05-18 00:13:18,238 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463530397439_0001_00_000004 is valid, but no longer registered, and will be killed
2016-05-18 00:13:18,238 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463530397439_0001_00_000004: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:13:18,238 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463530397439_0001_00_000004, Exiting
2016-05-18 00:13:18,238 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463530397439_0001_00_000004
2016-05-18 00:13:18,238 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463530397439_0001_00_000004
2016-05-18 00:13:18,239 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463530397439_0001_00_000004 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:13:18,239 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463530397439_0001_00_000004 completed successfully
2016-05-18 00:13:18,241 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:81BA6267A60E47359D72F741451CE540 of vertex/vertexGroup:F247AA1E0CE0490AA51BBE85178E228A isVertexGroupOutput:false
2016-05-18 00:13:18,241 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=4, numSuccessfulVertices=4, numFailedVertices=0, numKilledVertices=0, numVertices=4, commitInProgress=0, terminationCause=null
2016-05-18 00:13:18,245 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463530397439_0001_1, queueSize=0
2016-05-18 00:13:18,245 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463530397439_0001_1][Event:DAG_FINISHED]: dagId=dag_1463530397439_0001_1, startTime=1463530397646, finishTime=1463530398242, timeTaken=596, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=4, TOTAL_LAUNCHED_TASKS=4, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=1270, AM_GC_TIME_MILLIS=187, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=114, REDUCE_INPUT_RECORDS=296, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=592, NUM_SHUFFLED_INPUTS=3, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=3, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=317, OUTPUT_BYTES=21854, OUTPUT_BYTES_WITH_OVERHEAD=13723, OUTPUT_BYTES_PHYSICAL=13735, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=13735, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=3, SHUFFLE_BYTES=13735, SHUFFLE_BYTES_DECOMPRESSED=13723, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=13735, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=26, MERGE_PHASE_TIME=221, FIRST_EVENT_RECEIVED=14, LAST_EVENT_RECEIVED=14, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=5854121591711, Process_Duration=75, Process_End_Time=5854121591786, Read_Duration=24, Tuples_Read=303, Tuples_Written=317, Write_Duration=11, cascading.flow.StepCounters, Tuples_Read=7, Tuples_Written=21
2016-05-18 00:13:18,245 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463530397439_0001_1 finished with state: SUCCEEDED
2016-05-18 00:13:18,245 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463530397439_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:13:18 Completed Dag: dag_1463530397439_0001_1
2016-05-18 00:13:18,246 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463530397439_0001_1, dagState=SUCCEEDED
2016-05-18 00:13:18,246 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463530397439_0001_1_post
2016-05-18 00:13:18,246 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:13:18,246 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:13:18,246 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:13:18,246 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, with id=dag_1463530397439_0001_1
2016-05-18 00:13:18,247 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709] euclidean/(1/1) ...anceShort/euclidean/short, with id=dag_1463530397439_0001_1
2016-05-18 00:13:18,255 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...anceShort/euclidean/short, applicationId=application_1463530397439_0001
2016-05-18 00:13:18,257 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:13:18,257 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:13:18,258 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:13:18,258 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:13:18,258 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:13:18,258 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:13:18,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:13:18,260 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:13:18,260 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:13:18,260 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:13:18,260 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:13:18,262 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:13:18,262 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:13:18,263 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:13:18,266 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/39272CC47A804D6E848939B60FA3181A/BF91AB846325402A8E1453390790D709/.tez/application_1463530397439_0001
2016-05-18 00:13:18,267 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short/_temporary
2016-05-18 00:13:18,268 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [euclidean] completed in: 00:00.859
2016-05-18 00:13:18,357 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/distanceusecaseplatform/testEuclideanDistanceShort/euclidean/short