2016-05-18 01:47:43,446 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-18 01:47:43,478 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.SortedValuesPlatformTest, with platform: hadoop2-tez
2016-05-18 01:47:43,519 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-18 01:47:44,075 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-18 01:47:44,494 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-18 01:47:44,822 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:44,825 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:44,913 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: B593C02774B1424D9B0568890F260040
2016-05-18 01:47:45,173 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.126
2016-05-18 01:47:45,211 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.163
2016-05-18 01:47:45,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:45,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:45,215 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:45,390 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt
2016-05-18 01:47:45,574 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt
2016-05-18 01:47:45,627 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed
2016-05-18 01:47:45,917 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-18 01:47:45,919 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] starting
2016-05-18 01:47:45,920 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:45,920 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:45,921 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed"]
2016-05-18 01:47:45,921 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] parallel execution of steps is enabled: false
2016-05-18 01:47:45,921 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executing total steps: 1
2016-05-18 01:47:45,922 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] allocating management threads: 1
2016-05-18 01:47:45,923 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] starting step: (1/1) ...ed/cogroupcomparereversed
2016-05-18 01:47:45,923 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [apache*ip] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:45,941 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tez session mode enabled: true
2016-05-18 01:47:46,114 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:46,121 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:46,121 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:46,171 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001 doesn't exist and is created
2016-05-18 01:47:46,248 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:46,249 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:46,250 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd
2016-05-18 01:47:46,316 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536066120_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:46,330 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:46,331 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:46,340 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:46,340 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:46,340 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:46,341 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:46,349 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 01:47:46,349 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 01:47:46,349 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:46,364 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:46,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:46,389 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:46,391 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:46,392 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:46,392 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:46,393 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:46,394 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:46,395 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:46,396 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:46,396 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:46,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:46,412 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:46,430 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:46,433 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:46,433 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:46,436 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-log-dir
2016-05-18 01:47:46,437 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-log-dir/history.txt.appattempt_1463536066120_0001_000000, maxErrors=10
2016-05-18 01:47:46,437 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:46,439 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536066120_0001_000000, appSubmitTime=1463536066297, launchTime=1463536066314
2016-05-18 01:47:46,446 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:46,450 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-18 01:47:46,450 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-18 01:47:46,450 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:46,500 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42260
2016-05-18 01:47:46,502 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536066120
2016-05-18 01:47:46,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536066120_0001_000000, startTime=1463536066520
2016-05-18 01:47:46,521 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:46,550 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:46,566 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:46,566 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ed/cogroupcomparereversed, applicationId=application_1463536066120_0001, dagName=[CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed
2016-05-18 01:47:46,818 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed
2016-05-18 01:47:46,828 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:46,891 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536066120_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-log-dir/dag_1463536066120_0001_1.dot
2016-05-18 01:47:46,895 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-log-dir/dag_1463536066120_0001_1-tez-dag.pb.txt
2016-05-18 01:47:46,947 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536066120_0001_1
2016-05-18 01:47:46,947 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:46,947 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed
2016-05-18 01:47:46 Running Dag: dag_1463536066120_0001_1
2016-05-18 01:47:46,982 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536066120_0001_1, submitTime=1463536066818
2016-05-18 01:47:47,030 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001/tez-conf.pb" } size: 40937 timestamp: 1463536066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:47,031 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,031 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:47,034 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001/tez-conf.pb" } size: 40937 timestamp: 1463536066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:47,034 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,034 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:47,037 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001/tez-conf.pb" } size: 40937 timestamp: 1463536066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:47,037 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:47,038 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:47,047 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:47,049 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536066120_0001_1, initTime=1463536067049
2016-05-18 01:47:47,050 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536066120_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:47,052 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:47,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:DAG_STARTED]: dagID=dag_1463536066120_0001_1, startTime=1463536067054
2016-05-18 01:47:47,055 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536066120_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:47,055 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E736427781F842BD81DA59CC773026C9 : {620637C606204BCCA1204D412EC9FBA2={InputName=620637C606204BCCA1204D412EC9FBA2}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:47,056 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 620637C606204BCCA1204D412EC9FBA2, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:47,056 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,061 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ed/cogroupcomparereversed, applicationId=application_1463536066120_0001, dagName=[CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed
2016-05-18 01:47:47,063 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,063 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,065 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,071 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 620637C606204BCCA1204D412EC9FBA2 on vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,084 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:47,084 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F59EF03B4FCB4210A69943EF04155061 : {A08741F388144B3D8F0E5BFDA7A838CA={InputName=A08741F388144B3D8F0E5BFDA7A838CA}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:47,084 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A08741F388144B3D8F0E5BFDA7A838CA, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:47,085 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,085 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,085 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,086 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,086 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:47,087 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:47,094 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] submitted tez dag to app master: application_1463536066120_0001, with dag id: dag_1463536066120_0001_1
2016-05-18 01:47:47,100 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 620637C606204BCCA1204D412EC9FBA2 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:47,101 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A08741F388144B3D8F0E5BFDA7A838CA on vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,118 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A08741F388144B3D8F0E5BFDA7A838CA asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:47,129 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:47,130 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:47,147 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:47,148 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:47,149 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:47,150 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 32599 Min splitLength: 52428800 New desired splits: 1 Total length: 32599 Original splits: 2
2016-05-18 01:47:47,150 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 275 Min splitLength: 52428800 New desired splits: 1 Total length: 275 Original splits: 2
2016-05-18 01:47:47,151 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 32599 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 32599 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:47,151 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 275 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 275 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:47,153 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:47,156 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536066120_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=2, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:47,156 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69], numAdditionalOutputs=1
2016-05-18 01:47:47,157 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=CEFBCFA6F060488FA1DA9FE1B408EA1E, vertex=vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:47,152 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:47,180 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 620637C606204BCCA1204D412EC9FBA2 on vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,180 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A08741F388144B3D8F0E5BFDA7A838CA on vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,184 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FB4E86F018054978B566A04B8C033B69:CEFBCFA6F060488FA1DA9FE1B408EA1E using oldmapred API
2016-05-18 01:47:47,185 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=CEFBCFA6F060488FA1DA9FE1B408EA1E, vertexName=FB4E86F018054978B566A04B8C033B69, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:47,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FB4E86F018054978B566A04B8C033B69, vertexId=vertex_1463536066120_0001_1_02, initRequestedTime=1463536067087, initedTime=1463536067156, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:47,196 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:47,218 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:47,219 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A08741F388144B3D8F0E5BFDA7A838CA=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:47,219 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:47,220 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {620637C606204BCCA1204D412EC9FBA2=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:47,220 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] parallelism set to 1
2016-05-18 01:47:47,221 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] parallelism set to 1
2016-05-18 01:47:47,222 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:47,221 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:47,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F59EF03B4FCB4210A69943EF04155061, vertexId=vertex_1463536066120_0001_1_01, initRequestedTime=1463536067084, initedTime=1463536067229, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:47,230 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] with distanceFromRoot: 0
2016-05-18 01:47:47,230 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:47,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536066120_0001_1_01, startRequestedTime=1463536067087, startedTime=1463536067230
2016-05-18 01:47:47,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536066120_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:47,233 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:47,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E736427781F842BD81DA59CC773026C9, vertexId=vertex_1463536066120_0001_1_00, initRequestedTime=1463536067055, initedTime=1463536067233, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:47,233 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] with distanceFromRoot: 0
2016-05-18 01:47:47,233 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:47,234 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536066120_0001_1_01 for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] numStartedSources: 1 numSources: 2
2016-05-18 01:47:47,234 INFO impl.VertexImpl (VertexImpl.java:transition(2937)) - Cannot start vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] numStartedSources: 1 numSources: 2
2016-05-18 01:47:47,234 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536066120_0001_1_00, startRequestedTime=1463536067087, startedTime=1463536067234
2016-05-18 01:47:47,234 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536066120_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:47,235 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:47,235 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536066120_0001_1_00 for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] numStartedSources: 2 numSources: 2
2016-05-18 01:47:47,235 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] with distanceFromRoot: 1
2016-05-18 01:47:47,238 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F59EF03B4FCB4210A69943EF04155061
2016-05-18 01:47:47,238 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,240 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536066120_0001_1_02, startRequestedTime=1463536067235, startedTime=1463536067235
2016-05-18 01:47:47,240 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:47,241 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E736427781F842BD81DA59CC773026C9
2016-05-18 01:47:47,252 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,251 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_STARTED]: vertexName=F59EF03B4FCB4210A69943EF04155061, taskId=task_1463536066120_0001_1_01_000000, scheduledTime=1463536067250, launchTime=1463536067250
2016-05-18 01:47:47,261 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FB4E86F018054978B566A04B8C033B69 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:47,265 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E736427781F842BD81DA59CC773026C9 in vertex: FB4E86F018054978B566A04B8C033B69 numBipartiteSourceTasks: 1
2016-05-18 01:47:47,272 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F59EF03B4FCB4210A69943EF04155061 in vertex: FB4E86F018054978B566A04B8C033B69 numBipartiteSourceTasks: 2
2016-05-18 01:47:47,291 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_STARTED]: vertexName=E736427781F842BD81DA59CC773026C9, taskId=task_1463536066120_0001_1_00_000000, scheduledTime=1463536067291, launchTime=1463536067291
2016-05-18 01:47:47,294 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:47,295 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:47,338 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:47,355 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536066120_0001_00_000002, launchTime=1463536067355
2016-05-18 01:47:47,356 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536066120_0001_00_000001, launchTime=1463536067355
2016-05-18 01:47:47,361 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000002
2016-05-18 01:47:47,361 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000001
2016-05-18 01:47:47,365 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,365 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,367 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536066120_0001_1_00_000000_0] started. Is using containerId: [container_1463536066120_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:47 Starting to run new task attempt: attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,368 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,369 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:47,370 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:47,373 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E736427781F842BD81DA59CC773026C9, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0, startTime=1463536067365, containerId=container_1463536066120_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:47,374 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536066120_0001_1_01_000000_0] started. Is using containerId: [container_1463536066120_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:47,376 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F59EF03B4FCB4210A69943EF04155061, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0, startTime=1463536067374, containerId=container_1463536066120_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:47,376 INFO counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-18 01:47:47 Starting to run new task attempt: attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,383 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,383 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:47,384 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:47,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed, VertexName: F59EF03B4FCB4210A69943EF04155061, VertexParallelism: 1, TaskAttemptID:attempt_1463536066120_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A08741F388144B3D8F0E5BFDA7A838CA, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FB4E86F018054978B566A04B8C033B69, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:47,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed, VertexName: E736427781F842BD81DA59CC773026C9, VertexParallelism: 1, TaskAttemptID:attempt_1463536066120_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=620637C606204BCCA1204D412EC9FBA2, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FB4E86F018054978B566A04B8C033B69, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:47,399 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:47,399 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:47,405 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,405 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,433 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536066120_0001_1_01_000000_0 sent events: (0-1).
2016-05-18 01:47:47,434 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536066120_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:47,436 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536066120_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:47,437 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536066120_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:47,454 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:47,455 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:47,474 INFO input.MRInput (MRInput.java:initialize(440)) - 620637C606204BCCA1204D412EC9FBA2 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:47,475 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 620637C606204BCCA1204D412EC9FBA2
2016-05-18 01:47:47,480 INFO input.MRInput (MRInput.java:initialize(440)) - A08741F388144B3D8F0E5BFDA7A838CA using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:47,481 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A08741F388144B3D8F0E5BFDA7A838CA
2016-05-18 01:47:47,498 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F59EF03B4FCB4210A69943EF04155061, ordinal: 1
2016-05-18 01:47:47,499 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F59EF03B4FCB4210A69943EF04155061, mem on start (mb), free: 189, total: 281, max: 672
2016-05-18 01:47:47,499 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:47,499 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:47,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:47,501 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:47,501 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A08741F388144B3D8F0E5BFDA7A838CA:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FB4E86F018054978B566A04B8C033B69:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:47,504 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E736427781F842BD81DA59CC773026C9, ordinal: 0
2016-05-18 01:47:47,504 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E736427781F842BD81DA59CC773026C9, mem on start (mb), free: 189, total: 281, max: 672
2016-05-18 01:47:47,504 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:47,505 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:47,505 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:47,505 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:47,505 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[620637C606204BCCA1204D412EC9FBA2:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FB4E86F018054978B566A04B8C033B69:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:47,510 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:47,511 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:47,516 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:47,517 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:47,518 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:47,518 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A08741F388144B3D8F0E5BFDA7A838CA:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FB4E86F018054978B566A04B8C033B69:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:47,519 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:47,521 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A08741F388144B3D8F0E5BFDA7A838CA being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:47,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A08741F388144B3D8F0E5BFDA7A838CA
2016-05-18 01:47:47,522 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:47,522 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[620637C606204BCCA1204D412EC9FBA2:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FB4E86F018054978B566A04B8C033B69:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:47,523 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:47,523 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 620637C606204BCCA1204D412EC9FBA2
2016-05-18 01:47:47,523 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 620637C606204BCCA1204D412EC9FBA2 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:47,524 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:47,524 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:47,524 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:47,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:47,526 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:47,526 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:47,528 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,529 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,559 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt:0+137, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt:137+138], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=275}
2016-05-18 01:47:47,559 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt:0+16299, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt:16299+16300], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=32599}
2016-05-18 01:47:47,562 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 620637C606204BCCA1204D412EC9FBA2 initialized RecordReader from event
2016-05-18 01:47:47,562 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A08741F388144B3D8F0E5BFDA7A838CA initialized RecordReader from event
2016-05-18 01:47:47,576 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:47,576 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:47,621 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"] streamed: true, id: A08741F388144B3D8F0E5BFDA7A838CA
2016-05-18 01:47:47,621 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']], id: EF7E3B8C4273426A93DEC00C139A8A87
2016-05-18 01:47:47,621 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"] streamed: true, id: 620637C606204BCCA1204D412EC9FBA2
2016-05-18 01:47:47,622 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']], id: EF7E3B8C4273426A93DEC00C139A8A87
2016-05-18 01:47:47,622 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] EF7E3B8C4273426A93DEC00C139A8A87
2016-05-18 01:47:47,623 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] EF7E3B8C4273426A93DEC00C139A8A87
2016-05-18 01:47:47,679 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FB4E86F018054978B566A04B8C033B69 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@5afb8f1c, comparator=cascading.tuple.hadoop.util.TupleComparator@129dc9b8, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:47,679 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FB4E86F018054978B566A04B8C033B69 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3970a5ee, comparator=cascading.tuple.hadoop.util.TupleComparator@52e0efd6, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:47,690 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FB4E86F018054978B566A04B8C033B69: , UsingHashComparator=false
2016-05-18 01:47:47,690 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FB4E86F018054978B566A04B8C033B69: , UsingHashComparator=false
2016-05-18 01:47:47,798 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:47,798 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:47,800 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FB4E86F018054978B566A04B8C033B69: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:47,805 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FB4E86F018054978B566A04B8C033B69: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:47,813 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:47,818 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F59EF03B4FCB4210A69943EF04155061, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:47:47,907 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:47,908 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:47,908 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FB4E86F018054978B566A04B8C033B69: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:47,908 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FB4E86F018054978B566A04B8C033B69: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:47,910 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:47,911 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E736427781F842BD81DA59CC773026C9, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:47,916 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,917 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E736427781F842BD81DA59CC773026C9, mem on close (mb), free: 76, total: 382, max: 672
2016-05-18 01:47:47,917 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FB4E86F018054978B566A04B8C033B69: Starting flush of map output
2016-05-18 01:47:47,918 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FB4E86F018054978B566A04B8C033B69: Span0.length = 20, perItem = 5
2016-05-18 01:47:47,921 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FB4E86F018054978B566A04B8C033B69: done sorting span=0, length=20, time=3
2016-05-18 01:47:47,925 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FB4E86F018054978B566A04B8C033B69: Heap = SpanIterator<0:19> (span=Span[16777216,102]),
2016-05-18 01:47:47,933 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FB4E86F018054978B566A04B8C033B69: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-local-dir/output/attempt_1463536066120_0001_1_00_000000_0_10005_0/file.out
2016-05-18 01:47:47,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:47 Completed running task attempt: attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:47,975 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:47,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536066120_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:47,979 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:47,979 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F59EF03B4FCB4210A69943EF04155061, mem on close (mb), free: 72, total: 382, max: 672
2016-05-18 01:47:47,980 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FB4E86F018054978B566A04B8C033B69: Starting flush of map output
2016-05-18 01:47:47,980 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FB4E86F018054978B566A04B8C033B69: Span0.length = 200, perItem = 45
2016-05-18 01:47:47,981 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536066120_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536067911, Process_Duration=5, Process_End_Time=1463536067916, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 01:47:47,989 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536066120_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:47,990 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E736427781F842BD81DA59CC773026C9, taskAttemptId=attempt_1463536066120_0001_1_00_000000_0, creationTime=1463536067292, allocationTime=1463536067347, startTime=1463536067365, finishTime=1463536067988, timeTaken=623, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:47,990 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:47,991 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:47,991 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:47,991 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000002
2016-05-18 01:47:47,999 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536066120_0001_1_00_000000_0
2016-05-18 01:47:48,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_FINISHED]: vertexName=E736427781F842BD81DA59CC773026C9, taskId=task_1463536066120_0001_1_00_000000, startTime=1463536067365, finishTime=1463536068001, timeTaken=636, status=SUCCEEDED, successfulAttemptID=attempt_1463536066120_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536067911, Process_Duration=5, Process_End_Time=1463536067916, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 01:47:48,026 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:48,026 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:48,028 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FB4E86F018054978B566A04B8C033B69: done sorting span=0, length=200, time=48
2016-05-18 01:47:48,029 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FB4E86F018054978B566A04B8C033B69: Heap = SpanIterator<0:199> (span=Span[16777216,9061]),
2016-05-18 01:47:48,031 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FB4E86F018054978B566A04B8C033B69: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-local-dir/output/attempt_1463536066120_0001_1_01_000000_0_10006_0/file.out
2016-05-18 01:47:48,038 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_FINISHED]: vertexName=E736427781F842BD81DA59CC773026C9, vertexId=vertex_1463536066120_0001_1_00, initRequestedTime=1463536067055, initedTime=1463536067233, startRequestedTime=1463536067087, startedTime=1463536067234, finishTime=1463536068026, timeTaken=792, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536067911, Process_Duration=5, Process_End_Time=1463536067916, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463536067365, firstTasksToStart=[ task_1463536066120_0001_1_00_000000 ], lastTaskFinishTime=1463536067988, lastTasksToFinish=[ task_1463536066120_0001_1_00_000000 ], minTaskDuration=623, maxTaskDuration=623, avgTaskDuration=623.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536066120_0001_1_00_000000 ], longestDurationTasks=[ task_1463536066120_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:48,038 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:48,043 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536066120_0001_00_000002, stoppedTime=1463536068043, exitStatus=0
2016-05-18 01:47:48,044 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] attempt: attempt_1463536066120_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:48,045 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:48,046 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:48,046 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:48 Completed running task attempt: attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:48,050 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:48,050 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536066120_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:48,050 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536066120_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:48,051 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536066120_0001_1_01_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536067818, Process_Duration=161, Process_End_Time=1463536067979, Read_Duration=92, Tuples_Read=200, Tuples_Written=200, Write_Duration=20][cascading.flow.StepCounters Tuples_Read=200]]
2016-05-18 01:47:48,055 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536066120_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:48,055 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:48,056 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536066120_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:48,057 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000002: 65 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:48,057 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536066120_0001_00_000002, Exiting
2016-05-18 01:47:48,057 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536066120_0001_00_000002
2016-05-18 01:47:48,057 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536066120_0001_00_000002
2016-05-18 01:47:48,056 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:48,059 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:48,058 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F59EF03B4FCB4210A69943EF04155061, taskAttemptId=attempt_1463536066120_0001_1_01_000000_0, creationTime=1463536067286, allocationTime=1463536067346, startTime=1463536067374, finishTime=1463536068055, timeTaken=681, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:48,060 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536066120_0001_1_01_000000_0
2016-05-18 01:47:48,060 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000001
2016-05-18 01:47:48,065 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_FINISHED]: vertexName=F59EF03B4FCB4210A69943EF04155061, taskId=task_1463536066120_0001_1_01_000000, startTime=1463536067374, finishTime=1463536068060, timeTaken=686, status=SUCCEEDED, successfulAttemptID=attempt_1463536066120_0001_1_01_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536067818, Process_Duration=161, Process_End_Time=1463536067979, Read_Duration=92, Tuples_Read=200, Tuples_Written=200, Write_Duration=20, cascading.flow.StepCounters, Tuples_Read=200
2016-05-18 01:47:48,065 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536066120_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:48,066 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536066120_0001_00_000002 completed successfully
2016-05-18 01:47:48,067 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:48,068 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:48,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_FINISHED]: vertexName=F59EF03B4FCB4210A69943EF04155061, vertexId=vertex_1463536066120_0001_1_01, initRequestedTime=1463536067084, initedTime=1463536067229, startRequestedTime=1463536067087, startedTime=1463536067230, finishTime=1463536068068, timeTaken=838, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536067818, Process_Duration=161, Process_End_Time=1463536067979, Read_Duration=92, Tuples_Read=200, Tuples_Written=200, Write_Duration=20, cascading.flow.StepCounters, Tuples_Read=200, vertexStats=firstTaskStartTime=1463536067374, firstTasksToStart=[ task_1463536066120_0001_1_01_000000 ], lastTaskFinishTime=1463536068055, lastTasksToFinish=[ task_1463536066120_0001_1_01_000000 ], minTaskDuration=681, maxTaskDuration=681, avgTaskDuration=681.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536066120_0001_1_01_000000 ], longestDurationTasks=[ task_1463536066120_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:48,071 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:48,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536066120_0001_00_000001, stoppedTime=1463536068072, exitStatus=0
2016-05-18 01:47:48,072 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] attempt: attempt_1463536066120_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:48,072 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FB4E86F018054978B566A04B8C033B69
2016-05-18 01:47:48,073 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E736427781F842BD81DA59CC773026C9, destinationVertexName=FB4E86F018054978B566A04B8C033B69. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:48,073 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F59EF03B4FCB4210A69943EF04155061, destinationVertexName=FB4E86F018054978B566A04B8C033B69. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:48,074 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:48,075 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:48,075 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:48,075 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536066120_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:48,075 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_STARTED]: vertexName=FB4E86F018054978B566A04B8C033B69, taskId=task_1463536066120_0001_1_02_000000, scheduledTime=1463536068075, launchTime=1463536068075
2016-05-18 01:47:48,075 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:48,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536066120_0001_00_000003, launchTime=1463536068077
2016-05-18 01:47:48,078 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000003
2016-05-18 01:47:48,079 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536066120_0001_1_02_000000_0] started. Is using containerId: [container_1463536066120_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 01:47:48,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FB4E86F018054978B566A04B8C033B69, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0, startTime=1463536068079, containerId=container_1463536066120_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 01:47:48,080 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48 Starting to run new task attempt: attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,082 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,082 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:48,082 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:48,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed, VertexName: FB4E86F018054978B566A04B8C033B69, VertexParallelism: 1, TaskAttemptID:attempt_1463536066120_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E736427781F842BD81DA59CC773026C9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=F59EF03B4FCB4210A69943EF04155061, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=CEFBCFA6F060488FA1DA9FE1B408EA1E, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:48,083 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=2, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:48,084 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,087 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536066120_0001_1_02_000000_0 2 events [0,2) total 2 vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:48,088 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536066120_0001_1_02_000000_0, eventCount=2 fromEventId=0 nextFromEventId=2
2016-05-18 01:47:48,091 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536066120_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:48,091 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:48,091 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000001: 31 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:48,092 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536066120_0001_00_000001, Exiting
2016-05-18 01:47:48,092 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536066120_0001_00_000001
2016-05-18 01:47:48,092 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536066120_0001_00_000001
2016-05-18 01:47:48,093 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536066120_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:48,093 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536066120_0001_00_000001 completed successfully
2016-05-18 01:47:48,122 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FB4E86F018054978B566A04B8C033B69, ordinal: 2
2016-05-18 01:47:48,122 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FB4E86F018054978B566A04B8C033B69, mem on start (mb), free: 58, total: 382, max: 672
2016-05-18 01:47:48,122 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:48,122 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 3 initializers to finish
2016-05-18 01:47:48,126 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:48,126 INFO output.MROutput (MROutput.java:initialize(421)) - CEFBCFA6F060488FA1DA9FE1B408EA1E: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:48,128 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:48,128 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:48,129 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:48,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:48,130 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CEFBCFA6F060488FA1DA9FE1B408EA1E:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [F59EF03B4FCB4210A69943EF04155061:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [E736427781F842BD81DA59CC773026C9:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:48,130 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:48,130 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.045, finalReserveFractionUsed=0.345
2016-05-18 01:47:48,130 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 3, numScaledRequests: 25, TotalRequested: 422785824, TotalRequestedScaled: 2.0293719551999998E8, TotalJVMHeap: 704643072, TotalAvailable: 461541212, TotalRequested/TotalJVMHeap:0.60
2016-05-18 01:47:48,131 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CEFBCFA6F060488FA1DA9FE1B408EA1E:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [F59EF03B4FCB4210A69943EF04155061:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [E736427781F842BD81DA59CC773026C9:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:48,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:48,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E736427781F842BD81DA59CC773026C9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:48,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F59EF03B4FCB4210A69943EF04155061 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:48,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 2
2016-05-18 01:47:48,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 2 IOs to start
2016-05-18 01:47:48,132 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F59EF03B4FCB4210A69943EF04155061: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:48,133 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E736427781F842BD81DA59CC773026C9: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:48,135 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E736427781F842BD81DA59CC773026C9: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:48,135 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F59EF03B4FCB4210A69943EF04155061: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:48,155 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F59EF03B4FCB4210A69943EF04155061 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:48,155 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E736427781F842BD81DA59CC773026C9 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:48,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E736427781F842BD81DA59CC773026C9
2016-05-18 01:47:48,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:48,166 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F59EF03B4FCB4210A69943EF04155061
2016-05-18 01:47:48,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:48,167 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,171 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E736427781F842BD81DA59CC773026C9: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:48,175 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']]
2016-05-18 01:47:48,176 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F59EF03B4FCB4210A69943EF04155061: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:48,182 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536066120_0001_1_01_000000_0_10006} to DISK_DIRECT, csize=8931, dsize=8927, EndTime=1463536068182, TimeTaken=5, Rate=1.70 MB/s
2016-05-18 01:47:48,182 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F59EF03B4FCB4210A69943EF04155061
2016-05-18 01:47:48,182 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 01:47:48,183 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F59EF03B4FCB4210A69943EF04155061, wasInterrupted=false
2016-05-18 01:47:48,183 WARN util.TupleHasher (TupleHasher.java:merge(73)) - two unequal Hasher instances for the same key field position found: cascading.platform.hadoop.TestLongComparator@424c8abf, and: cascading.platform.hadoop.TestLongComparator@2f29239a
2016-05-18 01:47:48,184 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 01:47:48,184 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] streamed: true, id: EF7E3B8C4273426A93DEC00C139A8A87
2016-05-18 01:47:48,184 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:48,184 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536066120_0001_1_00_000000_0_10005} to DISK_DIRECT, csize=142, dsize=138, EndTime=1463536068184, TimeTaken=5, Rate=0.03 MB/s
2016-05-18 01:47:48,219 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E736427781F842BD81DA59CC773026C9
2016-05-18 01:47:48,219 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:48,219 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-local-dir/output/attempt_1463536066120_0001_1_01_000000_0_10006/file.out, len=8931, isLocal=true
2016-05-18 01:47:48,218 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed"], id: CEFBCFA6F060488FA1DA9FE1B408EA1E
2016-05-18 01:47:48,219 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E736427781F842BD81DA59CC773026C9, wasInterrupted=false
2016-05-18 01:47:48,221 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:48,221 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed"]
2016-05-18 01:47:48,221 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:48,223 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 8931 bytes from disk
2016-05-18 01:47:48,224 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:48,227 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] EF7E3B8C4273426A93DEC00C139A8A87, for 2 inputs
2016-05-18 01:47:48,229 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] EF7E3B8C4273426A93DEC00C139A8A87, for 2 inputs
2016-05-18 01:47:48,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001_wd/localmode-local-dir/output/attempt_1463536066120_0001_1_00_000000_0_10005/file.out, len=142, isLocal=true
2016-05-18 01:47:48,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 142 bytes from disk
2016-05-18 01:47:48,230 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:48,230 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:48,230 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:48,238 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 128 bytes
2016-05-18 01:47:48,242 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:48,242 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:48,239 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 8917 bytes
2016-05-18 01:47:48,243 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F59EF03B4FCB4210A69943EF04155061
2016-05-18 01:47:48,242 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E736427781F842BD81DA59CC773026C9
2016-05-18 01:47:48,244 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F59EF03B4FCB4210A69943EF04155061: Shuffle Runner thread complete
2016-05-18 01:47:48,244 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E736427781F842BD81DA59CC773026C9: Shuffle Runner thread complete
2016-05-18 01:47:48,248 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FB4E86F018054978B566A04B8C033B69, all 2 inputs ready in: 00:00:00.000
2016-05-18 01:47:48,250 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F59EF03B4FCB4210A69943EF04155061: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:48,256 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E736427781F842BD81DA59CC773026C9: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:48,300 INFO output.MROutput (MROutput.java:close(526)) - CEFBCFA6F060488FA1DA9FE1B408EA1E closed
2016-05-18 01:47:48,301 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536066120_0001_1_02_000000_0 given a go for committing the task output.
2016-05-18 01:47:48,302 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,303 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FB4E86F018054978B566A04B8C033B69, mem on close (mb), free: 152, total: 382, max: 672
2016-05-18 01:47:48,303 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E736427781F842BD81DA59CC773026C9
2016-05-18 01:47:48,303 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E736427781F842BD81DA59CC773026C9: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:48,304 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F59EF03B4FCB4210A69943EF04155061
2016-05-18 01:47:48,304 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F59EF03B4FCB4210A69943EF04155061: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:48,305 INFO output.MROutput (MROutput.java:close(526)) - CEFBCFA6F060488FA1DA9FE1B408EA1E closed
2016-05-18 01:47:48,305 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:48 Completed running task attempt: attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,307 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0, askedToStop=false
2016-05-18 01:47:48,307 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536066120_0001_1_02_000000_0, stopRequested=false
2016-05-18 01:47:48,308 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536066120_0001_1_02_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=139, MERGE_PHASE_TIME=220, FIRST_EVENT_RECEIVED=82, LAST_EVENT_RECEIVED=82][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536068249, Process_Duration=53, Process_End_Time=1463536068302, Read_Duration=4, Tuples_Read=124, Tuples_Written=199, Write_Duration=3][cascading.flow.StepCounters Tuples_Written=199]]
2016-05-18 01:47:48,310 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536066120_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:48,310 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FB4E86F018054978B566A04B8C033B69, taskAttemptId=attempt_1463536066120_0001_1_02_000000_0, creationTime=1463536068075, allocationTime=1463536068076, startTime=1463536068079, finishTime=1463536068310, timeTaken=231, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:48,310 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:48,311 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:48,311 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536066120_0001_1_02_000000_0
2016-05-18 01:47:48,311 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:48,315 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536066120_0001_00_000003
2016-05-18 01:47:48,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:TASK_FINISHED]: vertexName=FB4E86F018054978B566A04B8C033B69, taskId=task_1463536066120_0001_1_02_000000, startTime=1463536068079, finishTime=1463536068311, timeTaken=232, status=SUCCEEDED, successfulAttemptID=attempt_1463536066120_0001_1_02_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=139, MERGE_PHASE_TIME=220, FIRST_EVENT_RECEIVED=82, LAST_EVENT_RECEIVED=82, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536068249, Process_Duration=53, Process_End_Time=1463536068302, Read_Duration=4, Tuples_Read=124, Tuples_Written=199, Write_Duration=3, cascading.flow.StepCounters, Tuples_Written=199
2016-05-18 01:47:48,315 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:48,315 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69]
2016-05-18 01:47:48,320 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:VERTEX_FINISHED]: vertexName=FB4E86F018054978B566A04B8C033B69, vertexId=vertex_1463536066120_0001_1_02, initRequestedTime=1463536067087, initedTime=1463536067156, startRequestedTime=1463536067235, startedTime=1463536067235, finishTime=1463536068316, timeTaken=1081, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=139, MERGE_PHASE_TIME=220, FIRST_EVENT_RECEIVED=82, LAST_EVENT_RECEIVED=82, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536068249, Process_Duration=53, Process_End_Time=1463536068302, Read_Duration=4, Tuples_Read=124, Tuples_Written=199, Write_Duration=3, cascading.flow.StepCounters, Tuples_Written=199, vertexStats=firstTaskStartTime=1463536068079, firstTasksToStart=[ task_1463536066120_0001_1_02_000000 ], lastTaskFinishTime=1463536068310, lastTasksToFinish=[ task_1463536066120_0001_1_02_000000 ], minTaskDuration=231, maxTaskDuration=231, avgTaskDuration=231.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536066120_0001_1_02_000000 ], longestDurationTasks=[ task_1463536066120_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:48,320 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:48,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536066120_0001_00_000003, stoppedTime=1463536068321, exitStatus=0
2016-05-18 01:47:48,322 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:48,322 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:48,322 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536066120_0001_1_00 [E736427781F842BD81DA59CC773026C9]
2016-05-18 01:47:48,323 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536066120_0001_1_01 [F59EF03B4FCB4210A69943EF04155061]
2016-05-18 01:47:48,324 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536066120_0001_1
2016-05-18 01:47:48,325 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536066120_0001_1
2016-05-18 01:47:48,326 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536066120_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:48,326 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536066120_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 01:47:48,327 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536066120_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 01:47:48,326 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: CEFBCFA6F060488FA1DA9FE1B408EA1E for vertex: vertex_1463536066120_0001_1_02 [FB4E86F018054978B566A04B8C033B69], outputName: CEFBCFA6F060488FA1DA9FE1B408EA1E
2016-05-18 01:47:48,328 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536066120_0001_00_000003: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:48,330 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536066120_0001_00_000003, Exiting
2016-05-18 01:47:48,331 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536066120_0001_00_000003
2016-05-18 01:47:48,331 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536066120_0001_00_000003
2016-05-18 01:47:48,331 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536066120_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:48,332 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536066120_0001_00_000003 completed successfully
2016-05-18 01:47:48,347 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:CEFBCFA6F060488FA1DA9FE1B408EA1E of vertex/vertexGroup:FB4E86F018054978B566A04B8C033B69 isVertexGroupOutput:false
2016-05-18 01:47:48,347 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:48,357 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536066120_0001_1, queueSize=0
2016-05-18 01:47:48,359 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536066120_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536066120_0001_1, startTime=1463536067052, finishTime=1463536068347, timeTaken=1295, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=3, TOTAL_LAUNCHED_TASKS=3, RACK_LOCAL_TASKS=2, AM_CPU_MILLISECONDS=3170, AM_GC_TIME_MILLIS=32, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=440, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=419, OUTPUT_BYTES=9163, OUTPUT_BYTES_WITH_OVERHEAD=9065, OUTPUT_BYTES_PHYSICAL=9073, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=2, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=139, MERGE_PHASE_TIME=220, FIRST_EVENT_RECEIVED=82, LAST_EVENT_RECEIVED=82, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=4390608203978, Process_Duration=219, Process_End_Time=4390608204197, Read_Duration=98, Tuples_Read=344, Tuples_Written=419, Write_Duration=26, cascading.flow.StepCounters, Tuples_Read=220, Tuples_Written=199
2016-05-18 01:47:48,360 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536066120_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:48,362 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536066120_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:48 Completed Dag: dag_1463536066120_0001_1
2016-05-18 01:47:48,364 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536066120_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:48,365 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536066120_0001_1_post
2016-05-18 01:47:48,366 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:48,366 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:48,366 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:48,368 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed, with id=dag_1463536066120_0001_1
2016-05-18 01:47:48,370 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99] apache*ip/(1/1) ...ed/cogroupcomparereversed, with id=dag_1463536066120_0001_1
2016-05-18 01:47:48,376 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ed/cogroupcomparereversed, applicationId=application_1463536066120_0001
2016-05-18 01:47:48,379 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:48,379 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:48,379 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:48,379 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:48,380 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:48,382 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:48,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:48,384 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:48,384 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:48,385 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:48,385 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:48,386 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:48,387 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:48,388 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:48,392 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/CB2CF69540954E91BA4C08976D7AEBD0/A057272874EF4ECF8B87C051B6E13E99/.tez/application_1463536066120_0001
2016-05-18 01:47:48,395 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed/_temporary
2016-05-18 01:47:48,395 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] completed in: 00:02.474
2016-05-18 01:47:48,490 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValuesReversed/cogroupcomparereversed
2016-05-18 01:47:48,538 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:48,539 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:48,630 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.051
2016-05-18 01:47:48,675 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.096
2016-05-18 01:47:48,676 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:48,676 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:48,677 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:48,700 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt
2016-05-18 01:47:48,718 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt
2016-05-18 01:47:48,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward
2016-05-18 01:47:48,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] starting
2016-05-18 01:47:48,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:48,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:48,868 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward"]
2016-05-18 01:47:48,869 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] parallel execution of steps is enabled: false
2016-05-18 01:47:48,869 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] executing total steps: 1
2016-05-18 01:47:48,869 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] allocating management threads: 1
2016-05-18 01:47:48,870 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] starting step: (1/1) ...ues/cogroupcompareforward
2016-05-18 01:47:48,870 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [apache*ip] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:48,890 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] tez session mode enabled: true
2016-05-18 01:47:48,894 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:48,895 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:48,895 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:48,909 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001 doesn't exist and is created
2016-05-18 01:47:48,937 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:48,937 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:48,938 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd
2016-05-18 01:47:48,957 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536068895_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:48,957 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:48,957 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:48,959 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:48,959 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:48,960 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:48,960 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:48,961 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:48,964 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:48,965 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:48,965 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:48,965 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:48,965 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:48,966 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:48,966 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:48,966 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:48,966 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:48,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:48,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:48,967 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:48,972 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:48,974 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:48,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:48,975 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-log-dir
2016-05-18 01:47:48,975 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-log-dir/history.txt.appattempt_1463536068895_0001_000000, maxErrors=10
2016-05-18 01:47:48,976 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:48,976 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536068895_0001_000000, appSubmitTime=1463536068956, launchTime=1463536068956
2016-05-18 01:47:48,989 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:48,993 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60397
2016-05-18 01:47:48,994 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536068895
2016-05-18 01:47:48,999 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536068895_0001_000000, startTime=1463536068999
2016-05-18 01:47:48,999 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:49,037 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:49,038 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:49,038 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ues/cogroupcompareforward, applicationId=application_1463536068895_0001, dagName=[2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward
2016-05-18 01:47:49,084 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward
2016-05-18 01:47:49,085 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:49,086 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536068895_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-log-dir/dag_1463536068895_0001_1.dot
2016-05-18 01:47:49,088 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-log-dir/dag_1463536068895_0001_1-tez-dag.pb.txt
2016-05-18 01:47:49,101 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536068895_0001_1
2016-05-18 01:47:49,101 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,102 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward
2016-05-18 01:47:49 Running Dag: dag_1463536068895_0001_1
2016-05-18 01:47:49,119 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536068895_0001_1, submitTime=1463536069084
2016-05-18 01:47:49,120 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001/tez-conf.pb" } size: 40937 timestamp: 1463536068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:49,121 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,121 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:49,121 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001/tez-conf.pb" } size: 40937 timestamp: 1463536068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:49,122 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,122 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:49,122 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001/tez-conf.pb" } size: 40937 timestamp: 1463536068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:49,122 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,123 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:49,123 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:49,124 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536068895_0001_1, initTime=1463536069123
2016-05-18 01:47:49,124 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536068895_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:49,125 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:49,125 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:DAG_STARTED]: dagID=dag_1463536068895_0001_1, startTime=1463536069125
2016-05-18 01:47:49,125 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536068895_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:49,125 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B45CFE4D2AD14801813848DCD5A9634E : {5DAD16B0E271499981790F7C139EC02F={InputName=5DAD16B0E271499981790F7C139EC02F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:49,126 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 5DAD16B0E271499981790F7C139EC02F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:49,126 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,126 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,126 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,126 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,126 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ues/cogroupcompareforward, applicationId=application_1463536068895_0001, dagName=[2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F2611D50F2C642399E41BEEF5DA4C63D : {A33E9BE89B7243B5898B8FE9545A7190={InputName=A33E9BE89B7243B5898B8FE9545A7190}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A33E9BE89B7243B5898B8FE9545A7190, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,127 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,128 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,128 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:49,128 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,130 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 5DAD16B0E271499981790F7C139EC02F on vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,134 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 5DAD16B0E271499981790F7C139EC02F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:49,135 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:49,135 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,135 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A33E9BE89B7243B5898B8FE9545A7190 on vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,136 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536068895_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=2, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:49,139 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142], numAdditionalOutputs=1
2016-05-18 01:47:49,139 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=93B7904ACE2D434B8A4E30F0E820CA02, vertex=vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:49,143 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A33E9BE89B7243B5898B8FE9545A7190 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:49,151 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FA9DB5B3170449448F935CD72DC9A142:93B7904ACE2D434B8A4E30F0E820CA02 using oldmapred API
2016-05-18 01:47:49,151 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=93B7904ACE2D434B8A4E30F0E820CA02, vertexName=FA9DB5B3170449448F935CD72DC9A142, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:49,153 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:49,155 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 32599 Min splitLength: 52428800 New desired splits: 1 Total length: 32599 Original splits: 2
2016-05-18 01:47:49,155 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 32599 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 32599 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:49,155 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:49,154 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:49,156 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 275 Min splitLength: 52428800 New desired splits: 1 Total length: 275 Original splits: 2
2016-05-18 01:47:49,157 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 275 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 275 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:49,157 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] submitted tez dag to app master: application_1463536068895_0001, with dag id: dag_1463536068895_0001_1
2016-05-18 01:47:49,155 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, vertexId=vertex_1463536068895_0001_1_02, initRequestedTime=1463536069128, initedTime=1463536069139, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:49,159 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:49,157 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 5DAD16B0E271499981790F7C139EC02F on vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,157 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:49,163 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A33E9BE89B7243B5898B8FE9545A7190 on vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,163 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:49,165 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {5DAD16B0E271499981790F7C139EC02F=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:49,165 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:49,167 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A33E9BE89B7243B5898B8FE9545A7190=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:49,167 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] parallelism set to 1
2016-05-18 01:47:49,167 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:49,168 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] parallelism set to 1
2016-05-18 01:47:49,168 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:49,170 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, vertexId=vertex_1463536068895_0001_1_01, initRequestedTime=1463536069125, initedTime=1463536069170, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:49,170 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] with distanceFromRoot: 0
2016-05-18 01:47:49,171 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:49,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536068895_0001_1_01, startRequestedTime=1463536069128, startedTime=1463536069171
2016-05-18 01:47:49,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536068895_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:49,171 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:49,171 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536068895_0001_1_01 for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] numStartedSources: 1 numSources: 2
2016-05-18 01:47:49,171 INFO impl.VertexImpl (VertexImpl.java:transition(2937)) - Cannot start vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] numStartedSources: 1 numSources: 2
2016-05-18 01:47:49,173 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B45CFE4D2AD14801813848DCD5A9634E
2016-05-18 01:47:49,173 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,175 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_STARTED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, taskId=task_1463536068895_0001_1_01_000000, scheduledTime=1463536069174, launchTime=1463536069174
2016-05-18 01:47:49,176 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, vertexId=vertex_1463536068895_0001_1_00, initRequestedTime=1463536069127, initedTime=1463536069176, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:49,177 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] with distanceFromRoot: 0
2016-05-18 01:47:49,177 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:49,177 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536068895_0001_1_00, startRequestedTime=1463536069128, startedTime=1463536069178
2016-05-18 01:47:49,179 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536068895_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:49,179 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:49,179 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F2611D50F2C642399E41BEEF5DA4C63D
2016-05-18 01:47:49,180 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,180 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536068895_0001_1_00 for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] numStartedSources: 2 numSources: 2
2016-05-18 01:47:49,180 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] with distanceFromRoot: 1
2016-05-18 01:47:49,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536068895_0001_1_02, startRequestedTime=1463536069180, startedTime=1463536069181
2016-05-18 01:47:49,182 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:49,182 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:49,183 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FA9DB5B3170449448F935CD72DC9A142 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:49,184 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F2611D50F2C642399E41BEEF5DA4C63D in vertex: FA9DB5B3170449448F935CD72DC9A142 numBipartiteSourceTasks: 1
2016-05-18 01:47:49,184 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_STARTED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, taskId=task_1463536068895_0001_1_00_000000, scheduledTime=1463536069184, launchTime=1463536069184
2016-05-18 01:47:49,187 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B45CFE4D2AD14801813848DCD5A9634E in vertex: FA9DB5B3170449448F935CD72DC9A142 numBipartiteSourceTasks: 2
2016-05-18 01:47:49,188 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,194 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536068895_0001_00_000001, launchTime=1463536069194
2016-05-18 01:47:49,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536068895_0001_00_000002, launchTime=1463536069197
2016-05-18 01:47:49,197 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000001
2016-05-18 01:47:49,198 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000002
2016-05-18 01:47:49,199 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536068895_0001_1_01_000000_0] started. Is using containerId: [container_1463536068895_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:49,200 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,200 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0, startTime=1463536069198, containerId=container_1463536068895_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:49,201 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,201 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536068895_0001_1_00_000000_0] started. Is using containerId: [container_1463536068895_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:49,202 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0, startTime=1463536069201, containerId=container_1463536068895_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:49 Starting to run new task attempt: attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,203 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,203 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,203 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:49,204 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward, VertexName: F2611D50F2C642399E41BEEF5DA4C63D, VertexParallelism: 1, TaskAttemptID:attempt_1463536068895_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A33E9BE89B7243B5898B8FE9545A7190, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FA9DB5B3170449448F935CD72DC9A142, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:49,204 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:49,206 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49 Starting to run new task attempt: attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,211 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,211 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,211 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:49,211 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward, VertexName: B45CFE4D2AD14801813848DCD5A9634E, VertexParallelism: 1, TaskAttemptID:attempt_1463536068895_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=5DAD16B0E271499981790F7C139EC02F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FA9DB5B3170449448F935CD72DC9A142, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:49,211 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:49,220 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,221 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536068895_0001_1_01_000000_0 sent events: (0-1).
2016-05-18 01:47:49,223 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536068895_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:49,220 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536068895_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:49,224 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536068895_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:49,224 INFO input.MRInput (MRInput.java:initialize(440)) - A33E9BE89B7243B5898B8FE9545A7190 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:49,224 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A33E9BE89B7243B5898B8FE9545A7190
2016-05-18 01:47:49,228 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:49,230 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:49,243 INFO input.MRInput (MRInput.java:initialize(440)) - 5DAD16B0E271499981790F7C139EC02F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:49,243 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 5DAD16B0E271499981790F7C139EC02F
2016-05-18 01:47:49,246 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F2611D50F2C642399E41BEEF5DA4C63D, ordinal: 0
2016-05-18 01:47:49,247 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2611D50F2C642399E41BEEF5DA4C63D, mem on start (mb), free: 229, total: 490, max: 672
2016-05-18 01:47:49,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:49,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:49,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:49,249 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:49,250 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A33E9BE89B7243B5898B8FE9545A7190:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FA9DB5B3170449448F935CD72DC9A142:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:49,250 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:49,251 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:49,252 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:49,252 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A33E9BE89B7243B5898B8FE9545A7190:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FA9DB5B3170449448F935CD72DC9A142:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:49,252 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:49,253 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A33E9BE89B7243B5898B8FE9545A7190 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:49,253 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A33E9BE89B7243B5898B8FE9545A7190
2016-05-18 01:47:49,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:49,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:49,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:49,255 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,264 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:49,265 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B45CFE4D2AD14801813848DCD5A9634E, ordinal: 1
2016-05-18 01:47:49,266 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B45CFE4D2AD14801813848DCD5A9634E, mem on start (mb), free: 227, total: 490, max: 672
2016-05-18 01:47:49,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:49,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:49,265 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"] streamed: true, id: A33E9BE89B7243B5898B8FE9545A7190
2016-05-18 01:47:49,268 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']], id: 3F107D30997C465BB25CBE90CF6431D3
2016-05-18 01:47:49,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:49,269 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:49,269 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[5DAD16B0E271499981790F7C139EC02F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FA9DB5B3170449448F935CD72DC9A142:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:49,267 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt:0+137, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt:137+138], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=275}
2016-05-18 01:47:49,271 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A33E9BE89B7243B5898B8FE9545A7190 initialized RecordReader from event
2016-05-18 01:47:49,270 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:49,269 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] 3F107D30997C465BB25CBE90CF6431D3
2016-05-18 01:47:49,272 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:49,274 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:49,274 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[5DAD16B0E271499981790F7C139EC02F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FA9DB5B3170449448F935CD72DC9A142:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:49,274 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:49,276 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 5DAD16B0E271499981790F7C139EC02F
2016-05-18 01:47:49,276 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 5DAD16B0E271499981790F7C139EC02F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:49,278 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FA9DB5B3170449448F935CD72DC9A142 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@e481e5a, comparator=cascading.tuple.hadoop.util.TupleComparator@20c0ae09, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:49,278 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:49,279 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:49,280 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:49,280 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FA9DB5B3170449448F935CD72DC9A142: , UsingHashComparator=false
2016-05-18 01:47:49,281 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,290 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt:0+16299, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt:16299+16300], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=32599}
2016-05-18 01:47:49,292 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 5DAD16B0E271499981790F7C139EC02F initialized RecordReader from event
2016-05-18 01:47:49,305 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:49,306 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:49,306 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FA9DB5B3170449448F935CD72DC9A142: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:49,307 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FA9DB5B3170449448F935CD72DC9A142: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:49,307 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/ips.20.txt"]
2016-05-18 01:47:49,308 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F2611D50F2C642399E41BEEF5DA4C63D, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:49,306 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:49,314 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"] streamed: true, id: 5DAD16B0E271499981790F7C139EC02F
2016-05-18 01:47:49,316 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']], id: 3F107D30997C465BB25CBE90CF6431D3
2016-05-18 01:47:49,316 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] 3F107D30997C465BB25CBE90CF6431D3
2016-05-18 01:47:49,315 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,317 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2611D50F2C642399E41BEEF5DA4C63D, mem on close (mb), free: 124, total: 490, max: 672
2016-05-18 01:47:49,320 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FA9DB5B3170449448F935CD72DC9A142: Starting flush of map output
2016-05-18 01:47:49,320 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FA9DB5B3170449448F935CD72DC9A142: Span0.length = 20, perItem = 5
2016-05-18 01:47:49,322 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FA9DB5B3170449448F935CD72DC9A142: done sorting span=0, length=20, time=2
2016-05-18 01:47:49,322 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FA9DB5B3170449448F935CD72DC9A142: Heap = SpanIterator<0:19> (span=Span[16777216,102]),
2016-05-18 01:47:49,324 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FA9DB5B3170449448F935CD72DC9A142: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-local-dir/output/attempt_1463536068895_0001_1_00_000000_0_10013_0/file.out
2016-05-18 01:47:49,326 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FA9DB5B3170449448F935CD72DC9A142 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@34ca81a4, comparator=cascading.tuple.hadoop.util.TupleComparator@22e1b60e, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:49,327 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FA9DB5B3170449448F935CD72DC9A142: , UsingHashComparator=false
2016-05-18 01:47:49,763 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:49,763 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:49,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:49,763 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FA9DB5B3170449448F935CD72DC9A142: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:49 Completed running task attempt: attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,764 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FA9DB5B3170449448F935CD72DC9A142: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:49,764 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:49,764 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536068895_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:49,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536068895_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536069308, Process_Duration=7, Process_End_Time=1463536069315, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 01:47:49,765 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/apache.200.txt"]
2016-05-18 01:47:49,765 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B45CFE4D2AD14801813848DCD5A9634E, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:49,768 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536068895_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,768 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,769 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:49,769 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,769 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, taskAttemptId=attempt_1463536068895_0001_1_00_000000_0, creationTime=1463536069185, allocationTime=1463536069194, startTime=1463536069201, finishTime=1463536069769, timeTaken=568, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:49,769 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000002
2016-05-18 01:47:49,771 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536068895_0001_1_00_000000_0
2016-05-18 01:47:49,772 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_FINISHED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, taskId=task_1463536068895_0001_1_00_000000, startTime=1463536069201, finishTime=1463536069771, timeTaken=570, status=SUCCEEDED, successfulAttemptID=attempt_1463536068895_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536069308, Process_Duration=7, Process_End_Time=1463536069315, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 01:47:49,772 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:49,773 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_FINISHED]: vertexName=F2611D50F2C642399E41BEEF5DA4C63D, vertexId=vertex_1463536068895_0001_1_00, initRequestedTime=1463536069127, initedTime=1463536069176, startRequestedTime=1463536069128, startedTime=1463536069178, finishTime=1463536069773, timeTaken=595, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=102, OUTPUT_BYTES_WITH_OVERHEAD=138, OUTPUT_BYTES_PHYSICAL=142, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536069308, Process_Duration=7, Process_End_Time=1463536069315, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463536069201, firstTasksToStart=[ task_1463536068895_0001_1_00_000000 ], lastTaskFinishTime=1463536069769, lastTasksToFinish=[ task_1463536068895_0001_1_00_000000 ], minTaskDuration=568, maxTaskDuration=568, avgTaskDuration=568.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536068895_0001_1_00_000000 ], longestDurationTasks=[ task_1463536068895_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:49,777 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:49,779 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536068895_0001_00_000002, stoppedTime=1463536069779, exitStatus=0
2016-05-18 01:47:49,781 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] attempt: attempt_1463536068895_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:49,780 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536068895_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:49,782 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000002: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:49,783 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536068895_0001_00_000002, Exiting
2016-05-18 01:47:49,783 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536068895_0001_00_000002
2016-05-18 01:47:49,783 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536068895_0001_00_000002
2016-05-18 01:47:49,782 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:49,785 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:49,786 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536068895_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:49,788 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536068895_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:49,788 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536068895_0001_00_000002 completed successfully
2016-05-18 01:47:49,815 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,815 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B45CFE4D2AD14801813848DCD5A9634E, mem on close (mb), free: 219, total: 440, max: 672
2016-05-18 01:47:49,816 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FA9DB5B3170449448F935CD72DC9A142: Starting flush of map output
2016-05-18 01:47:49,816 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FA9DB5B3170449448F935CD72DC9A142: Span0.length = 200, perItem = 45
2016-05-18 01:47:49,829 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FA9DB5B3170449448F935CD72DC9A142: done sorting span=0, length=200, time=13
2016-05-18 01:47:49,830 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FA9DB5B3170449448F935CD72DC9A142: Heap = SpanIterator<0:199> (span=Span[16777216,9061]),
2016-05-18 01:47:49,830 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FA9DB5B3170449448F935CD72DC9A142: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-local-dir/output/attempt_1463536068895_0001_1_01_000000_0_10016_0/file.out
2016-05-18 01:47:49,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:49 Completed running task attempt: attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,838 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:49,838 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536068895_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:49,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536068895_0001_1_01_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536069766, Process_Duration=49, Process_End_Time=1463536069815, Read_Duration=2, Tuples_Read=200, Tuples_Written=200, Write_Duration=23][cascading.flow.StepCounters Tuples_Read=200]]
2016-05-18 01:47:49,840 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536068895_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,840 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,840 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, taskAttemptId=attempt_1463536068895_0001_1_01_000000_0, creationTime=1463536069175, allocationTime=1463536069188, startTime=1463536069198, finishTime=1463536069840, timeTaken=642, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:49,841 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536068895_0001_1_01_000000_0
2016-05-18 01:47:49,840 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:49,841 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,841 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000001
2016-05-18 01:47:49,841 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_FINISHED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, taskId=task_1463536068895_0001_1_01_000000, startTime=1463536069198, finishTime=1463536069841, timeTaken=643, status=SUCCEEDED, successfulAttemptID=attempt_1463536068895_0001_1_01_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536069766, Process_Duration=49, Process_End_Time=1463536069815, Read_Duration=2, Tuples_Read=200, Tuples_Written=200, Write_Duration=23, cascading.flow.StepCounters, Tuples_Read=200
2016-05-18 01:47:49,841 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:49,843 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,844 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_FINISHED]: vertexName=B45CFE4D2AD14801813848DCD5A9634E, vertexId=vertex_1463536068895_0001_1_01, initRequestedTime=1463536069125, initedTime=1463536069170, startRequestedTime=1463536069128, startedTime=1463536069171, finishTime=1463536069843, timeTaken=672, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=200, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=200, OUTPUT_BYTES=9061, OUTPUT_BYTES_WITH_OVERHEAD=8927, OUTPUT_BYTES_PHYSICAL=8931, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536069766, Process_Duration=49, Process_End_Time=1463536069815, Read_Duration=2, Tuples_Read=200, Tuples_Written=200, Write_Duration=23, cascading.flow.StepCounters, Tuples_Read=200, vertexStats=firstTaskStartTime=1463536069198, firstTasksToStart=[ task_1463536068895_0001_1_01_000000 ], lastTaskFinishTime=1463536069840, lastTasksToFinish=[ task_1463536068895_0001_1_01_000000 ], minTaskDuration=642, maxTaskDuration=642, avgTaskDuration=642.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536068895_0001_1_01_000000 ], longestDurationTasks=[ task_1463536068895_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:49,844 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:49,844 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] attempt: attempt_1463536068895_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:49,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536068895_0001_00_000001, stoppedTime=1463536069845, exitStatus=0
2016-05-18 01:47:49,845 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FA9DB5B3170449448F935CD72DC9A142
2016-05-18 01:47:49,845 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B45CFE4D2AD14801813848DCD5A9634E, destinationVertexName=FA9DB5B3170449448F935CD72DC9A142. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:49,846 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F2611D50F2C642399E41BEEF5DA4C63D, destinationVertexName=FA9DB5B3170449448F935CD72DC9A142. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:49,846 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,847 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:49,847 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:49,847 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536068895_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:49,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_STARTED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, taskId=task_1463536068895_0001_1_02_000000, scheduledTime=1463536069847, launchTime=1463536069847
2016-05-18 01:47:49,848 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,851 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536068895_0001_00_000003, launchTime=1463536069850
2016-05-18 01:47:49,851 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000003
2016-05-18 01:47:49,851 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536068895_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:49,852 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:49,852 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536068895_0001_00_000001, Exiting
2016-05-18 01:47:49,852 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,852 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536068895_0001_1_02_000000_0] started. Is using containerId: [container_1463536068895_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 01:47:49,852 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536068895_0001_00_000001
2016-05-18 01:47:49,852 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536068895_0001_00_000001
2016-05-18 01:47:49,852 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0, startTime=1463536069852, containerId=container_1463536068895_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 01:47:49,854 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536068895_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:49,854 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536068895_0001_00_000001 completed successfully
2016-05-18 01:47:49 Starting to run new task attempt: attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,855 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,855 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,856 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:49,856 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward, VertexName: FA9DB5B3170449448F935CD72DC9A142, VertexParallelism: 1, TaskAttemptID:attempt_1463536068895_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B45CFE4D2AD14801813848DCD5A9634E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=F2611D50F2C642399E41BEEF5DA4C63D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=93B7904ACE2D434B8A4E30F0E820CA02, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:49,856 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=2, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:49,857 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,859 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536068895_0001_1_02_000000_0 2 events [0,2) total 2 vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,859 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536068895_0001_1_02_000000_0, eventCount=2 fromEventId=0 nextFromEventId=2
2016-05-18 01:47:49,862 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:49,862 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:49,869 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:49,881 INFO output.MROutput (MROutput.java:initialize(421)) - 93B7904ACE2D434B8A4E30F0E820CA02: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:49,885 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FA9DB5B3170449448F935CD72DC9A142, ordinal: 2
2016-05-18 01:47:49,885 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FA9DB5B3170449448F935CD72DC9A142, mem on start (mb), free: 211, total: 440, max: 672
2016-05-18 01:47:49,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:49,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 3 initializers to finish
2016-05-18 01:47:49,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:49,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:49,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:49,887 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[93B7904ACE2D434B8A4E30F0E820CA02:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B45CFE4D2AD14801813848DCD5A9634E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [F2611D50F2C642399E41BEEF5DA4C63D:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:49,887 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:49,888 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.045, finalReserveFractionUsed=0.345
2016-05-18 01:47:49,888 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 3, numScaledRequests: 25, TotalRequested: 422785824, TotalRequestedScaled: 2.0293719551999998E8, TotalJVMHeap: 704643072, TotalAvailable: 461541212, TotalRequested/TotalJVMHeap:0.60
2016-05-18 01:47:49,889 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[93B7904ACE2D434B8A4E30F0E820CA02:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B45CFE4D2AD14801813848DCD5A9634E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [F2611D50F2C642399E41BEEF5DA4C63D:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:49,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:49,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B45CFE4D2AD14801813848DCD5A9634E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:49,890 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B45CFE4D2AD14801813848DCD5A9634E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:49,890 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B45CFE4D2AD14801813848DCD5A9634E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:49,890 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F2611D50F2C642399E41BEEF5DA4C63D: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:49,890 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F2611D50F2C642399E41BEEF5DA4C63D: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:49,891 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B45CFE4D2AD14801813848DCD5A9634E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:49,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F2611D50F2C642399E41BEEF5DA4C63D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:49,893 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F2611D50F2C642399E41BEEF5DA4C63D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:49,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B45CFE4D2AD14801813848DCD5A9634E
2016-05-18 01:47:49,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 2
2016-05-18 01:47:49,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 2 IOs to start
2016-05-18 01:47:49,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:49,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F2611D50F2C642399E41BEEF5DA4C63D
2016-05-18 01:47:49,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:49,898 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,902 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F2611D50F2C642399E41BEEF5DA4C63D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:49,902 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B45CFE4D2AD14801813848DCD5A9634E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:49,906 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536068895_0001_1_00_000000_0_10013} to DISK_DIRECT, csize=142, dsize=138, EndTime=1463536069906, TimeTaken=4, Rate=0.03 MB/s
2016-05-18 01:47:49,906 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F2611D50F2C642399E41BEEF5DA4C63D
2016-05-18 01:47:49,906 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:49,906 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F2611D50F2C642399E41BEEF5DA4C63D, wasInterrupted=false
2016-05-18 01:47:49,908 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:49,909 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:49,910 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']]
2016-05-18 01:47:49,910 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-local-dir/output/attempt_1463536068895_0001_1_00_000000_0_10013/file.out, len=142, isLocal=true
2016-05-18 01:47:49,911 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 142 bytes from disk
2016-05-18 01:47:49,911 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:49,911 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:49,913 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 129 bytes
2016-05-18 01:47:49,913 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F2611D50F2C642399E41BEEF5DA4C63D
2016-05-18 01:47:49,913 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F2611D50F2C642399E41BEEF5DA4C63D: Shuffle Runner thread complete
2016-05-18 01:47:49,911 WARN util.TupleHasher (TupleHasher.java:merge(73)) - two unequal Hasher instances for the same key field position found: cascading.platform.hadoop.TestLongComparator@49705ef8, and: cascading.platform.hadoop.TestLongComparator@706b1172
2016-05-18 01:47:49,914 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - F2611D50F2C642399E41BEEF5DA4C63D: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:49,914 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] streamed: true, id: 3F107D30997C465BB25CBE90CF6431D3
2016-05-18 01:47:49,915 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536068895_0001_1_01_000000_0_10016} to DISK_DIRECT, csize=8931, dsize=8927, EndTime=1463536069915, TimeTaken=3, Rate=2.84 MB/s
2016-05-18 01:47:49,916 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B45CFE4D2AD14801813848DCD5A9634E
2016-05-18 01:47:49,916 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 01:47:49,917 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B45CFE4D2AD14801813848DCD5A9634E, wasInterrupted=false
2016-05-18 01:47:49,917 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.01 MB/s)
2016-05-18 01:47:49,917 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward"], id: 93B7904ACE2D434B8A4E30F0E820CA02
2016-05-18 01:47:49,918 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:49,918 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward"]
2016-05-18 01:47:49,919 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] 3F107D30997C465BB25CBE90CF6431D3, for 2 inputs
2016-05-18 01:47:49,919 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(apache*ip)[by: apache:[{1}:'octet'] ip:[{1}:'rawoctet']] 3F107D30997C465BB25CBE90CF6431D3, for 2 inputs
2016-05-18 01:47:49,920 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001_wd/localmode-local-dir/output/attempt_1463536068895_0001_1_01_000000_0_10016/file.out, len=8931, isLocal=true
2016-05-18 01:47:49,920 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 8931 bytes from disk
2016-05-18 01:47:49,920 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:49,920 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:49,920 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:49,920 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:49,921 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 8918 bytes
2016-05-18 01:47:49,921 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B45CFE4D2AD14801813848DCD5A9634E
2016-05-18 01:47:49,921 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B45CFE4D2AD14801813848DCD5A9634E: Shuffle Runner thread complete
2016-05-18 01:47:49,921 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FA9DB5B3170449448F935CD72DC9A142, all 2 inputs ready in: 00:00:00.001
2016-05-18 01:47:49,922 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B45CFE4D2AD14801813848DCD5A9634E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:49,925 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F2611D50F2C642399E41BEEF5DA4C63D: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:49,959 INFO output.MROutput (MROutput.java:close(526)) - 93B7904ACE2D434B8A4E30F0E820CA02 closed
2016-05-18 01:47:49,961 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536068895_0001_1_02_000000_0 given a go for committing the task output.
2016-05-18 01:47:49,962 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,962 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FA9DB5B3170449448F935CD72DC9A142, mem on close (mb), free: 206, total: 440, max: 672
2016-05-18 01:47:49,963 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B45CFE4D2AD14801813848DCD5A9634E
2016-05-18 01:47:49,963 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B45CFE4D2AD14801813848DCD5A9634E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:49,963 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F2611D50F2C642399E41BEEF5DA4C63D
2016-05-18 01:47:49,964 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F2611D50F2C642399E41BEEF5DA4C63D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:49,964 INFO output.MROutput (MROutput.java:close(526)) - 93B7904ACE2D434B8A4E30F0E820CA02 closed
2016-05-18 01:47:49,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:49 Completed running task attempt: attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,965 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0, askedToStop=false
2016-05-18 01:47:49,965 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536068895_0001_1_02_000000_0, stopRequested=false
2016-05-18 01:47:49,966 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536068895_0001_1_02_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=44, MERGE_PHASE_TIME=54, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536069921, Process_Duration=41, Process_End_Time=1463536069962, Read_Duration=7, Tuples_Read=124, Tuples_Written=199, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=199]]
2016-05-18 01:47:49,971 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536068895_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,971 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, taskAttemptId=attempt_1463536068895_0001_1_02_000000_0, creationTime=1463536069848, allocationTime=1463536069850, startTime=1463536069852, finishTime=1463536069971, timeTaken=119, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:49,973 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536068895_0001_1_02_000000_0
2016-05-18 01:47:49,971 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:49,973 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:49,973 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,973 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536068895_0001_00_000003
2016-05-18 01:47:49,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:TASK_FINISHED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, taskId=task_1463536068895_0001_1_02_000000, startTime=1463536069852, finishTime=1463536069973, timeTaken=121, status=SUCCEEDED, successfulAttemptID=attempt_1463536068895_0001_1_02_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=44, MERGE_PHASE_TIME=54, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536069921, Process_Duration=41, Process_End_Time=1463536069962, Read_Duration=7, Tuples_Read=124, Tuples_Written=199, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=199
2016-05-18 01:47:49,974 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:49,974 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142]
2016-05-18 01:47:49,976 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:VERTEX_FINISHED]: vertexName=FA9DB5B3170449448F935CD72DC9A142, vertexId=vertex_1463536068895_0001_1_02, initRequestedTime=1463536069128, initedTime=1463536069139, startRequestedTime=1463536069180, startedTime=1463536069181, finishTime=1463536069974, timeTaken=793, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=220, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=199, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=44, MERGE_PHASE_TIME=54, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536069921, Process_Duration=41, Process_End_Time=1463536069962, Read_Duration=7, Tuples_Read=124, Tuples_Written=199, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=199, vertexStats=firstTaskStartTime=1463536069852, firstTasksToStart=[ task_1463536068895_0001_1_02_000000 ], lastTaskFinishTime=1463536069971, lastTasksToFinish=[ task_1463536068895_0001_1_02_000000 ], minTaskDuration=119, maxTaskDuration=119, avgTaskDuration=119.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536068895_0001_1_02_000000 ], longestDurationTasks=[ task_1463536068895_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:49,976 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:49,976 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536068895_0001_00_000003, stoppedTime=1463536069976, exitStatus=0
2016-05-18 01:47:49,977 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:49,977 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:49,977 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536068895_0001_1_01 [B45CFE4D2AD14801813848DCD5A9634E]
2016-05-18 01:47:49,977 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536068895_0001_1_00 [F2611D50F2C642399E41BEEF5DA4C63D]
2016-05-18 01:47:49,977 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536068895_0001_1
2016-05-18 01:47:49,977 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536068895_0001_1
2016-05-18 01:47:49,978 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 93B7904ACE2D434B8A4E30F0E820CA02 for vertex: vertex_1463536068895_0001_1_02 [FA9DB5B3170449448F935CD72DC9A142], outputName: 93B7904ACE2D434B8A4E30F0E820CA02
2016-05-18 01:47:49,978 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536068895_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:49,980 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536068895_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 01:47:49,984 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536068895_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 01:47:49,984 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536068895_0001_00_000003: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:49,984 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536068895_0001_00_000003, Exiting
2016-05-18 01:47:49,984 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536068895_0001_00_000003
2016-05-18 01:47:49,984 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536068895_0001_00_000003
2016-05-18 01:47:49,987 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536068895_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:49,987 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536068895_0001_00_000003 completed successfully
2016-05-18 01:47:49,991 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:93B7904ACE2D434B8A4E30F0E820CA02 of vertex/vertexGroup:FA9DB5B3170449448F935CD72DC9A142 isVertexGroupOutput:false
2016-05-18 01:47:49,991 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:49,994 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536068895_0001_1, queueSize=0
2016-05-18 01:47:49,996 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536068895_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536068895_0001_1, startTime=1463536069125, finishTime=1463536069991, timeTaken=866, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=3, TOTAL_LAUNCHED_TASKS=3, RACK_LOCAL_TASKS=2, AM_CPU_MILLISECONDS=2070, AM_GC_TIME_MILLIS=422, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=69, REDUCE_INPUT_RECORDS=124, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=440, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=419, OUTPUT_BYTES=9163, OUTPUT_BYTES_WITH_OVERHEAD=9065, OUTPUT_BYTES_PHYSICAL=9073, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=9073, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=2, SHUFFLE_BYTES=9073, SHUFFLE_BYTES_DECOMPRESSED=9065, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=9073, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=44, MERGE_PHASE_TIME=54, FIRST_EVENT_RECEIVED=24, LAST_EVENT_RECEIVED=24, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=4390608208995, Process_Duration=97, Process_End_Time=4390608209092, Read_Duration=12, Tuples_Read=344, Tuples_Written=419, Write_Duration=24, cascading.flow.StepCounters, Tuples_Read=220, Tuples_Written=199
2016-05-18 01:47:49,996 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536068895_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:49,996 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536068895_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:49 Completed Dag: dag_1463536068895_0001_1
2016-05-18 01:47:49,998 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536068895_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:49,998 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536068895_0001_1_post
2016-05-18 01:47:49,998 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:49,999 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:49,999 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:49,999 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward, with id=dag_1463536068895_0001_1
2016-05-18 01:47:49,999 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816] apache*ip/(1/1) ...ues/cogroupcompareforward, with id=dag_1463536068895_0001_1
2016-05-18 01:47:50,004 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ues/cogroupcompareforward, applicationId=application_1463536068895_0001
2016-05-18 01:47:50,007 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:50,007 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:50,007 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:50,007 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:50,007 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:50,008 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:50,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:50,008 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:50,009 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:50,009 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:50,009 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:50,010 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:50,010 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:50,012 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:50,015 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2D4D1D365B90435AB9DBE40FB59A4877/F3B303B880C34E64A484AECB024E3816/.tez/application_1463536068895_0001
2016-05-18 01:47:50,016 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward/_temporary
2016-05-18 01:47:50,017 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache*ip] completed in: 00:01.149
2016-05-18 01:47:50,087 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testCoGroupComparatorValues/cogroupcompareforward
2016-05-18 01:47:50,128 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:50,128 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:50,189 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.012
2016-05-18 01:47:50,192 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.015
2016-05-18 01:47:50,193 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 01:47:50,193 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [apache] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 01:47:50,240 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:50,241 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:50,317 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.028
2016-05-18 01:47:50,331 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.042
2016-05-18 01:47:50,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:50,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:50,333 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:50,360 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:50,374 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false
2016-05-18 01:47:50,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:50,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:50,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false"]
2016-05-18 01:47:50,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:50,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:50,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:50,463 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ve/false_null_false_false
2016-05-18 01:47:50,463 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:50,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:50,491 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:50,492 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:50,492 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:50,502 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001 doesn't exist and is created
2016-05-18 01:47:50,530 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:50,530 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:50,531 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd
2016-05-18 01:47:50,551 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536070491_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:50,551 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:50,552 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:50,554 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:50,554 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:50,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:50,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:50,556 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:50,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:50,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:50,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:50,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:50,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:50,560 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:50,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:50,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:50,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:50,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:50,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:50,561 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:50,564 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:50,565 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:50,565 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:50,565 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-log-dir
2016-05-18 01:47:50,566 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-log-dir/history.txt.appattempt_1463536070491_0001_000000, maxErrors=10
2016-05-18 01:47:50,566 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:50,566 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536070491_0001_000000, appSubmitTime=1463536070550, launchTime=1463536070550
2016-05-18 01:47:50,573 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:50,915 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-18 01:47:50,915 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-18 01:47:50,915 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:50,915 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42060
2016-05-18 01:47:50,916 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536070491
2016-05-18 01:47:50,919 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536070491_0001_000000, startTime=1463536070919
2016-05-18 01:47:50,919 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:51,015 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:51,015 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:51,016 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ve/false_null_false_false, applicationId=application_1463536070491_0001, dagName=[B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false
2016-05-18 01:47:51,038 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false
2016-05-18 01:47:51,038 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:51,039 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536070491_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-log-dir/dag_1463536070491_0001_1.dot
2016-05-18 01:47:51,040 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-log-dir/dag_1463536070491_0001_1-tez-dag.pb.txt
2016-05-18 01:47:51,048 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536070491_0001_1
2016-05-18 01:47:51,048 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,048 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false
2016-05-18 01:47:51 Running Dag: dag_1463536070491_0001_1
2016-05-18 01:47:51,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536070491_0001_1, submitTime=1463536071038
2016-05-18 01:47:51,064 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001/tez-conf.pb" } size: 40937 timestamp: 1463536070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:51,064 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,064 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:51,066 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001/tez-conf.pb" } size: 40937 timestamp: 1463536070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:51,066 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,066 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:51,067 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:51,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536070491_0001_1, initTime=1463536071068
2016-05-18 01:47:51,069 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536070491_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:51,069 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:51,070 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:DAG_STARTED]: dagID=dag_1463536070491_0001_1, startTime=1463536071069
2016-05-18 01:47:51,070 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536070491_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C0B6F699235743168CCFCF704D2CC3C9 : {88748C854A57451880AC5FDC4A8456BC={InputName=88748C854A57451880AC5FDC4A8456BC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 88748C854A57451880AC5FDC4A8456BC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,070 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,071 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:51,071 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,072 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 88748C854A57451880AC5FDC4A8456BC on vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,074 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:51,074 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,074 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,074 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536070491_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:51,074 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A], numAdditionalOutputs=1
2016-05-18 01:47:51,080 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=43123343C46C4E4D8E786189BC6EA5C4, vertex=vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:51,082 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for BAA007D23ECC48A3A637361430C0466A:43123343C46C4E4D8E786189BC6EA5C4 using oldmapred API
2016-05-18 01:47:51,082 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ve/false_null_false_false, applicationId=application_1463536070491_0001, dagName=[B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false
2016-05-18 01:47:51,083 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 88748C854A57451880AC5FDC4A8456BC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:51,083 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=43123343C46C4E4D8E786189BC6EA5C4, vertexName=BAA007D23ECC48A3A637361430C0466A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:51,085 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BAA007D23ECC48A3A637361430C0466A, vertexId=vertex_1463536070491_0001_1_01, initRequestedTime=1463536071071, initedTime=1463536071074, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:51,085 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:51,086 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:51,086 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:51,087 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:51,087 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:51,087 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 88748C854A57451880AC5FDC4A8456BC on vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,089 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:51,089 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {88748C854A57451880AC5FDC4A8456BC=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:51,090 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] parallelism set to 1
2016-05-18 01:47:51,090 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:51,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, vertexId=vertex_1463536070491_0001_1_00, initRequestedTime=1463536071070, initedTime=1463536071091, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:51,091 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] with distanceFromRoot: 0
2016-05-18 01:47:51,092 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:51,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536070491_0001_1_00, startRequestedTime=1463536071071, startedTime=1463536071092
2016-05-18 01:47:51,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536070491_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:51,094 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:51,094 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C0B6F699235743168CCFCF704D2CC3C9
2016-05-18 01:47:51,094 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,094 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536070491_0001_1_00 for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] numStartedSources: 1 numSources: 1
2016-05-18 01:47:51,095 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] with distanceFromRoot: 1
2016-05-18 01:47:51,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536070491_0001_1_01, startRequestedTime=1463536071094, startedTime=1463536071095
2016-05-18 01:47:51,096 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:51,097 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_STARTED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, taskId=task_1463536070491_0001_1_00_000000, scheduledTime=1463536071096, launchTime=1463536071096
2016-05-18 01:47:51,097 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,098 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BAA007D23ECC48A3A637361430C0466A with 0 source tasks and 1 pending tasks
2016-05-18 01:47:51,099 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:51,100 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C0B6F699235743168CCFCF704D2CC3C9 in vertex: BAA007D23ECC48A3A637361430C0466A numBipartiteSourceTasks: 1
2016-05-18 01:47:51,105 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536070491_0001_00_000001, launchTime=1463536071105
2016-05-18 01:47:51,106 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536070491_0001_00_000001
2016-05-18 01:47:51,107 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536070491_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,107 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536070491_0001_1_00_000000_0] started. Is using containerId: [container_1463536070491_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:51,107 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0, startTime=1463536071107, containerId=container_1463536070491_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:51 Starting to run new task attempt: attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,108 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536070491_0001, with dag id: dag_1463536070491_0001_1
2016-05-18 01:47:51,108 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,109 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,109 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:51,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false, VertexName: C0B6F699235743168CCFCF704D2CC3C9, VertexParallelism: 1, TaskAttemptID:attempt_1463536070491_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=88748C854A57451880AC5FDC4A8456BC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BAA007D23ECC48A3A637361430C0466A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:51,110 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:51,112 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,116 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536070491_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:51,116 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536070491_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:51,120 INFO input.MRInput (MRInput.java:initialize(440)) - 88748C854A57451880AC5FDC4A8456BC using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:51,121 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:51,121 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 88748C854A57451880AC5FDC4A8456BC
2016-05-18 01:47:51,129 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C0B6F699235743168CCFCF704D2CC3C9, ordinal: 0
2016-05-18 01:47:51,129 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C0B6F699235743168CCFCF704D2CC3C9, mem on start (mb), free: 312, total: 357, max: 672
2016-05-18 01:47:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:51,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:51,130 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[88748C854A57451880AC5FDC4A8456BC:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BAA007D23ECC48A3A637361430C0466A:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:51,130 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:51,131 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:51,131 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:51,131 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[88748C854A57451880AC5FDC4A8456BC:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BAA007D23ECC48A3A637361430C0466A:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:51,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:51,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 88748C854A57451880AC5FDC4A8456BC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:51,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 88748C854A57451880AC5FDC4A8456BC
2016-05-18 01:47:51,132 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:51,132 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:51,132 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:51,133 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,137 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:51,137 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:51,137 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 88748C854A57451880AC5FDC4A8456BC initialized RecordReader from event
2016-05-18 01:47:51,139 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 88748C854A57451880AC5FDC4A8456BC
2016-05-18 01:47:51,139 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 7D16ADC2015248D3AED690D07B3225DE
2016-05-18 01:47:51,139 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7D16ADC2015248D3AED690D07B3225DE
2016-05-18 01:47:51,142 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BAA007D23ECC48A3A637361430C0466A using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6fd43c45, comparator=cascading.tuple.hadoop.util.TupleComparator@56c10f5e, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:51,142 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BAA007D23ECC48A3A637361430C0466A: , UsingHashComparator=false
2016-05-18 01:47:51,155 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:51,155 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:51,155 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BAA007D23ECC48A3A637361430C0466A: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:51,155 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BAA007D23ECC48A3A637361430C0466A: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:51,156 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:51,157 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C0B6F699235743168CCFCF704D2CC3C9, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:51,162 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,162 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C0B6F699235743168CCFCF704D2CC3C9, mem on close (mb), free: 210, total: 357, max: 672
2016-05-18 01:47:51,163 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BAA007D23ECC48A3A637361430C0466A: Starting flush of map output
2016-05-18 01:47:51,163 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BAA007D23ECC48A3A637361430C0466A: Span0.length = 37, perItem = 17
2016-05-18 01:47:51,165 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BAA007D23ECC48A3A637361430C0466A: done sorting span=0, length=37, time=2
2016-05-18 01:47:51,165 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BAA007D23ECC48A3A637361430C0466A: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:51,167 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BAA007D23ECC48A3A637361430C0466A: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-local-dir/output/attempt_1463536070491_0001_1_00_000000_0_10023_0/file.out
2016-05-18 01:47:51,170 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:51 Completed running task attempt: attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,171 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:51,171 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536070491_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:51,172 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536070491_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536071157, Process_Duration=5, Process_End_Time=1463536071162, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:51,173 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536070491_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,174 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, taskAttemptId=attempt_1463536070491_0001_1_00_000000_0, creationTime=1463536071097, allocationTime=1463536071103, startTime=1463536071107, finishTime=1463536071173, timeTaken=66, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:51,174 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,174 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:51,174 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,174 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536070491_0001_00_000001
2016-05-18 01:47:51,174 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536070491_0001_1_00_000000_0
2016-05-18 01:47:51,175 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_FINISHED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, taskId=task_1463536070491_0001_1_00_000000, startTime=1463536071107, finishTime=1463536071175, timeTaken=68, status=SUCCEEDED, successfulAttemptID=attempt_1463536070491_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536071157, Process_Duration=5, Process_End_Time=1463536071162, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:51,176 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:51,176 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_FINISHED]: vertexName=C0B6F699235743168CCFCF704D2CC3C9, vertexId=vertex_1463536070491_0001_1_00, initRequestedTime=1463536071070, initedTime=1463536071091, startRequestedTime=1463536071071, startedTime=1463536071092, finishTime=1463536071176, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536071157, Process_Duration=5, Process_End_Time=1463536071162, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536071107, firstTasksToStart=[ task_1463536070491_0001_1_00_000000 ], lastTaskFinishTime=1463536071173, lastTasksToFinish=[ task_1463536070491_0001_1_00_000000 ], minTaskDuration=66, maxTaskDuration=66, avgTaskDuration=66.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536070491_0001_1_00_000000 ], longestDurationTasks=[ task_1463536070491_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:51,178 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:51,179 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] attempt: attempt_1463536070491_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:51,179 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BAA007D23ECC48A3A637361430C0466A
2016-05-18 01:47:51,179 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C0B6F699235743168CCFCF704D2CC3C9, destinationVertexName=BAA007D23ECC48A3A637361430C0466A. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:51,179 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536070491_0001_00_000001, stoppedTime=1463536071179, exitStatus=0
2016-05-18 01:47:51,180 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:51,181 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_STARTED]: vertexName=BAA007D23ECC48A3A637361430C0466A, taskId=task_1463536070491_0001_1_01_000000, scheduledTime=1463536071181, launchTime=1463536071181
2016-05-18 01:47:51,181 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,181 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536070491_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:51,185 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536070491_0001_00_000002, launchTime=1463536071185
2016-05-18 01:47:51,186 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536070491_0001_00_000002
2016-05-18 01:47:51,186 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536070491_0001_1_01_000000_0] started. Is using containerId: [container_1463536070491_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:51,187 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BAA007D23ECC48A3A637361430C0466A, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0, startTime=1463536071186, containerId=container_1463536070491_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:51,187 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536070491_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,187 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536070491_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:51,187 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536070491_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:51 Starting to run new task attempt: attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,188 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536070491_0001_00_000001, Exiting
2016-05-18 01:47:51,188 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,188 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536070491_0001_00_000001
2016-05-18 01:47:51,188 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536070491_0001_00_000001
2016-05-18 01:47:51,188 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,188 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:51,189 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false, VertexName: BAA007D23ECC48A3A637361430C0466A, VertexParallelism: 1, TaskAttemptID:attempt_1463536070491_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C0B6F699235743168CCFCF704D2CC3C9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=43123343C46C4E4D8E786189BC6EA5C4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:51,189 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536070491_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:51,189 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:51,189 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536070491_0001_00_000001 completed successfully
2016-05-18 01:47:51,190 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,195 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536070491_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,196 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536070491_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:51,197 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:51,201 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:51,210 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BAA007D23ECC48A3A637361430C0466A, ordinal: 1
2016-05-18 01:47:51,210 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BAA007D23ECC48A3A637361430C0466A, mem on start (mb), free: 205, total: 357, max: 672
2016-05-18 01:47:51,210 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:51,211 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:51,211 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:51,211 INFO output.MROutput (MROutput.java:initialize(421)) - 43123343C46C4E4D8E786189BC6EA5C4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:51,212 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:51,213 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[43123343C46C4E4D8E786189BC6EA5C4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C0B6F699235743168CCFCF704D2CC3C9:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:51,213 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:51,214 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:51,214 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:51,214 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[43123343C46C4E4D8E786189BC6EA5C4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C0B6F699235743168CCFCF704D2CC3C9:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:51,214 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:51,214 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C0B6F699235743168CCFCF704D2CC3C9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:51,215 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C0B6F699235743168CCFCF704D2CC3C9: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:51,215 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C0B6F699235743168CCFCF704D2CC3C9: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:51,216 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:51,216 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:51,217 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C0B6F699235743168CCFCF704D2CC3C9 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:51,219 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C0B6F699235743168CCFCF704D2CC3C9
2016-05-18 01:47:51,219 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:51,220 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,220 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C0B6F699235743168CCFCF704D2CC3C9: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:51,224 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536070491_0001_1_00_000000_0_10023} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536071224, TimeTaken=1, Rate=0.57 MB/s
2016-05-18 01:47:51,224 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C0B6F699235743168CCFCF704D2CC3C9
2016-05-18 01:47:51,225 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:51,225 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C0B6F699235743168CCFCF704D2CC3C9, wasInterrupted=false
2016-05-18 01:47:51,226 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:51,227 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:51,227 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 7D16ADC2015248D3AED690D07B3225DE
2016-05-18 01:47:51,228 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false"], id: 43123343C46C4E4D8E786189BC6EA5C4
2016-05-18 01:47:51,228 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false"]
2016-05-18 01:47:51,229 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7D16ADC2015248D3AED690D07B3225DE, for 1 inputs
2016-05-18 01:47:51,228 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:51,231 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001_wd/localmode-local-dir/output/attempt_1463536070491_0001_1_00_000000_0_10023/file.out, len=593, isLocal=true
2016-05-18 01:47:51,232 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:51,232 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:51,232 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:51,233 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:51,233 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C0B6F699235743168CCFCF704D2CC3C9
2016-05-18 01:47:51,234 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C0B6F699235743168CCFCF704D2CC3C9: Shuffle Runner thread complete
2016-05-18 01:47:51,233 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BAA007D23ECC48A3A637361430C0466A, all 1 inputs ready in: 00:00:00.004
2016-05-18 01:47:51,235 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C0B6F699235743168CCFCF704D2CC3C9: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:51,241 INFO output.MROutput (MROutput.java:close(526)) - 43123343C46C4E4D8E786189BC6EA5C4 closed
2016-05-18 01:47:51,244 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536070491_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:51,244 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,245 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BAA007D23ECC48A3A637361430C0466A, mem on close (mb), free: 202, total: 357, max: 672
2016-05-18 01:47:51,245 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C0B6F699235743168CCFCF704D2CC3C9
2016-05-18 01:47:51,245 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C0B6F699235743168CCFCF704D2CC3C9: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:51,246 INFO output.MROutput (MROutput.java:close(526)) - 43123343C46C4E4D8E786189BC6EA5C4 closed
2016-05-18 01:47:51,246 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:51 Completed running task attempt: attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,247 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:51,247 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536070491_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:51,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536070491_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536071234, Process_Duration=10, Process_End_Time=1463536071244, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:51,249 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536070491_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,249 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,249 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BAA007D23ECC48A3A637361430C0466A, taskAttemptId=attempt_1463536070491_0001_1_01_000000_0, creationTime=1463536071181, allocationTime=1463536071184, startTime=1463536071186, finishTime=1463536071249, timeTaken=63, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:51,249 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:51,249 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,249 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536070491_0001_1_01_000000_0
2016-05-18 01:47:51,250 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536070491_0001_00_000002
2016-05-18 01:47:51,250 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:TASK_FINISHED]: vertexName=BAA007D23ECC48A3A637361430C0466A, taskId=task_1463536070491_0001_1_01_000000, startTime=1463536071186, finishTime=1463536071250, timeTaken=64, status=SUCCEEDED, successfulAttemptID=attempt_1463536070491_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536071234, Process_Duration=10, Process_End_Time=1463536071244, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:51,251 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:51,251 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A]
2016-05-18 01:47:51,252 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:VERTEX_FINISHED]: vertexName=BAA007D23ECC48A3A637361430C0466A, vertexId=vertex_1463536070491_0001_1_01, initRequestedTime=1463536071071, initedTime=1463536071074, startRequestedTime=1463536071094, startedTime=1463536071095, finishTime=1463536071252, timeTaken=157, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536071234, Process_Duration=10, Process_End_Time=1463536071244, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536071186, firstTasksToStart=[ task_1463536070491_0001_1_01_000000 ], lastTaskFinishTime=1463536071249, lastTasksToFinish=[ task_1463536070491_0001_1_01_000000 ], minTaskDuration=63, maxTaskDuration=63, avgTaskDuration=63.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536070491_0001_1_01_000000 ], longestDurationTasks=[ task_1463536070491_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:51,253 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:51,253 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536070491_0001_00_000002, stoppedTime=1463536071253, exitStatus=0
2016-05-18 01:47:51,253 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:51,253 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,253 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536070491_0001_1_00 [C0B6F699235743168CCFCF704D2CC3C9]
2016-05-18 01:47:51,253 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536070491_0001_1
2016-05-18 01:47:51,254 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536070491_0001_1
2016-05-18 01:47:51,254 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 43123343C46C4E4D8E786189BC6EA5C4 for vertex: vertex_1463536070491_0001_1_01 [BAA007D23ECC48A3A637361430C0466A], outputName: 43123343C46C4E4D8E786189BC6EA5C4
2016-05-18 01:47:51,255 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536070491_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:51,256 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536070491_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:51,261 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536070491_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:51,262 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536070491_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:51,262 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536070491_0001_00_000002, Exiting
2016-05-18 01:47:51,262 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536070491_0001_00_000002
2016-05-18 01:47:51,262 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536070491_0001_00_000002
2016-05-18 01:47:51,263 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536070491_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:51,263 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536070491_0001_00_000002 completed successfully
2016-05-18 01:47:51,264 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:43123343C46C4E4D8E786189BC6EA5C4 of vertex/vertexGroup:BAA007D23ECC48A3A637361430C0466A isVertexGroupOutput:false
2016-05-18 01:47:51,264 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,267 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536070491_0001_1, queueSize=0
2016-05-18 01:47:51,268 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536070491_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536070491_0001_1, startTime=1463536071069, finishTime=1463536071265, timeTaken=196, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=350, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072142391, Process_Duration=15, Process_End_Time=2927072142406, Read_Duration=5, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:51,268 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536070491_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:51,268 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536070491_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:51 Completed Dag: dag_1463536070491_0001_1
2016-05-18 01:47:51,269 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536070491_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:51,269 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536070491_0001_1_post
2016-05-18 01:47:51,269 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,269 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:51,269 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:51,270 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false, with id=dag_1463536070491_0001_1
2016-05-18 01:47:51,270 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7] comprehensivesort/(1/1) ...ve/false_null_false_false, with id=dag_1463536070491_0001_1
2016-05-18 01:47:51,276 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ve/false_null_false_false, applicationId=application_1463536070491_0001
2016-05-18 01:47:51,278 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:51,278 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:51,278 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:51,279 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:51,279 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:51,279 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:51,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:51,281 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:51,281 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:51,281 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:51,282 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:51,282 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:51,283 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:51,285 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:51,289 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B82B3D61124E49B993A1B16E7DC30209/EB5407301EBA40E6A40C4F6AD1A308D7/.tez/application_1463536070491_0001
2016-05-18 01:47:51,290 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false/_temporary
2016-05-18 01:47:51,290 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.828
2016-05-18 01:47:51,372 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:51,372 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:51,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.021
2016-05-18 01:47:51,440 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.036
2016-05-18 01:47:51,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:51,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:51,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:51,460 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:51,469 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false
2016-05-18 01:47:51,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:51,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:51,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false"]
2016-05-18 01:47:51,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:51,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:51,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:51,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/true_null_false_false
2016-05-18 01:47:51,529 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:51,544 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:51,548 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:51,549 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:51,549 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:51,558 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001 doesn't exist and is created
2016-05-18 01:47:51,578 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:51,578 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:51,579 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd
2016-05-18 01:47:51,595 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536071549_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:51,595 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:51,595 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:51,597 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:51,597 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:51,598 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:51,598 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:51,600 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:51,601 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:51,601 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:51,602 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:51,602 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:51,603 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:51,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:51,604 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:51,606 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:51,607 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:51,607 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:51,607 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-log-dir
2016-05-18 01:47:51,608 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-log-dir/history.txt.appattempt_1463536071549_0001_000000, maxErrors=10
2016-05-18 01:47:51,608 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:51,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536071549_0001_000000, appSubmitTime=1463536071594, launchTime=1463536071594
2016-05-18 01:47:51,622 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:51,625 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:33903
2016-05-18 01:47:51,625 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536071549
2016-05-18 01:47:51,627 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536071549_0001_000000, startTime=1463536071627
2016-05-18 01:47:51,627 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:51,679 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:51,679 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:51,679 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/true_null_false_false, applicationId=application_1463536071549_0001, dagName=[99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false
2016-05-18 01:47:51,699 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false
2016-05-18 01:47:51,700 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:51,701 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536071549_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-log-dir/dag_1463536071549_0001_1.dot
2016-05-18 01:47:51,702 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-log-dir/dag_1463536071549_0001_1-tez-dag.pb.txt
2016-05-18 01:47:51,709 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536071549_0001_1
2016-05-18 01:47:51,709 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,709 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false
2016-05-18 01:47:51 Running Dag: dag_1463536071549_0001_1
2016-05-18 01:47:51,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536071549_0001_1, submitTime=1463536071699
2016-05-18 01:47:51,727 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001/tez-conf.pb" } size: 40937 timestamp: 1463536071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:51,727 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,727 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:51,728 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001/tez-conf.pb" } size: 40937 timestamp: 1463536071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:51,729 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,730 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:51,730 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:51,731 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536071549_0001_1, initTime=1463536071731
2016-05-18 01:47:51,732 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536071549_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:51,733 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:51,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:DAG_STARTED]: dagID=dag_1463536071549_0001_1, startTime=1463536071733
2016-05-18 01:47:51,733 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/true_null_false_false, applicationId=application_1463536071549_0001, dagName=[99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false
2016-05-18 01:47:51,733 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536071549_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:51,734 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E2763F4F362E4CC3A26F9BB12E7ED979 : {795859061316494CBED026C4EB97A2CC={InputName=795859061316494CBED026C4EB97A2CC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:51,734 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 795859061316494CBED026C4EB97A2CC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:51,734 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,734 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,734 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,735 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,735 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:51,735 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 795859061316494CBED026C4EB97A2CC on vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,736 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,741 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:51,742 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,742 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,744 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536071549_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:51,744 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337], numAdditionalOutputs=1
2016-05-18 01:47:51,745 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=330A7B22BD2D45A187E9FA8FAD740BCC, vertex=vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:51,747 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 795859061316494CBED026C4EB97A2CC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:51,753 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DA3B2CDDB00D447992FCCB4219B72337:330A7B22BD2D45A187E9FA8FAD740BCC using oldmapred API
2016-05-18 01:47:51,753 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=330A7B22BD2D45A187E9FA8FAD740BCC, vertexName=DA3B2CDDB00D447992FCCB4219B72337, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:51,755 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, vertexId=vertex_1463536071549_0001_1_01, initRequestedTime=1463536071736, initedTime=1463536071744, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:51,756 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:51,756 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:51,756 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:51,757 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:51,757 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:51,758 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 795859061316494CBED026C4EB97A2CC on vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,759 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:51,760 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {795859061316494CBED026C4EB97A2CC=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:51,760 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] parallelism set to 1
2016-05-18 01:47:51,761 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:51,761 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, vertexId=vertex_1463536071549_0001_1_00, initRequestedTime=1463536071734, initedTime=1463536071761, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:51,761 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] with distanceFromRoot: 0
2016-05-18 01:47:51,762 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:51,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536071549_0001_1_00, startRequestedTime=1463536071736, startedTime=1463536071762
2016-05-18 01:47:51,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536071549_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:51,762 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:51,762 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536071549_0001_1_00 for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] numStartedSources: 1 numSources: 1
2016-05-18 01:47:51,762 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] with distanceFromRoot: 1
2016-05-18 01:47:51,764 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536071549_0001_1_01, startRequestedTime=1463536071762, startedTime=1463536071763
2016-05-18 01:47:51,765 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:51,765 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536071549_0001, with dag id: dag_1463536071549_0001_1
2016-05-18 01:47:51,769 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E2763F4F362E4CC3A26F9BB12E7ED979
2016-05-18 01:47:51,769 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_STARTED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, taskId=task_1463536071549_0001_1_00_000000, scheduledTime=1463536071771, launchTime=1463536071771
2016-05-18 01:47:51,773 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,774 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DA3B2CDDB00D447992FCCB4219B72337 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:51,774 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:51,779 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E2763F4F362E4CC3A26F9BB12E7ED979 in vertex: DA3B2CDDB00D447992FCCB4219B72337 numBipartiteSourceTasks: 1
2016-05-18 01:47:51,782 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536071549_0001_00_000001, launchTime=1463536071782
2016-05-18 01:47:51,783 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536071549_0001_00_000001
2016-05-18 01:47:51,783 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536071549_0001_1_00_000000_0] started. Is using containerId: [container_1463536071549_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:51,784 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0, startTime=1463536071783, containerId=container_1463536071549_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:51,784 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536071549_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51 Starting to run new task attempt: attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,785 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,785 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,785 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:51,786 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false, VertexName: E2763F4F362E4CC3A26F9BB12E7ED979, VertexParallelism: 1, TaskAttemptID:attempt_1463536071549_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=795859061316494CBED026C4EB97A2CC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DA3B2CDDB00D447992FCCB4219B72337, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:51,786 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:51,786 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,791 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536071549_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:51,791 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536071549_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:51,794 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:51,795 INFO input.MRInput (MRInput.java:initialize(440)) - 795859061316494CBED026C4EB97A2CC using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:51,796 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 795859061316494CBED026C4EB97A2CC
2016-05-18 01:47:51,801 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E2763F4F362E4CC3A26F9BB12E7ED979, ordinal: 0
2016-05-18 01:47:51,801 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E2763F4F362E4CC3A26F9BB12E7ED979, mem on start (mb), free: 305, total: 355, max: 672
2016-05-18 01:47:51,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:51,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:51,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:51,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:51,801 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[795859061316494CBED026C4EB97A2CC:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DA3B2CDDB00D447992FCCB4219B72337:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:51,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:51,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:51,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:51,802 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[795859061316494CBED026C4EB97A2CC:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DA3B2CDDB00D447992FCCB4219B72337:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 795859061316494CBED026C4EB97A2CC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 795859061316494CBED026C4EB97A2CC
2016-05-18 01:47:51,802 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:51,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,804 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:51,805 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 795859061316494CBED026C4EB97A2CC initialized RecordReader from event
2016-05-18 01:47:51,806 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:51,807 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 795859061316494CBED026C4EB97A2CC
2016-05-18 01:47:51,807 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 0EDFF2C128EB4B5F83CE057625A8BE39
2016-05-18 01:47:51,807 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 0EDFF2C128EB4B5F83CE057625A8BE39
2016-05-18 01:47:51,809 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DA3B2CDDB00D447992FCCB4219B72337 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2168212f, comparator=cascading.tuple.hadoop.util.TupleComparator@3c3ad0de, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:51,810 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DA3B2CDDB00D447992FCCB4219B72337: , UsingHashComparator=false
2016-05-18 01:47:51,822 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:51,822 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:51,822 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DA3B2CDDB00D447992FCCB4219B72337: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:51,822 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DA3B2CDDB00D447992FCCB4219B72337: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:51,823 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:51,823 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E2763F4F362E4CC3A26F9BB12E7ED979, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:51,830 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,830 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E2763F4F362E4CC3A26F9BB12E7ED979, mem on close (mb), free: 203, total: 355, max: 672
2016-05-18 01:47:51,831 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DA3B2CDDB00D447992FCCB4219B72337: Starting flush of map output
2016-05-18 01:47:51,831 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DA3B2CDDB00D447992FCCB4219B72337: Span0.length = 37, perItem = 17
2016-05-18 01:47:51,832 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DA3B2CDDB00D447992FCCB4219B72337: done sorting span=0, length=37, time=1
2016-05-18 01:47:51,833 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DA3B2CDDB00D447992FCCB4219B72337: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:51,834 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DA3B2CDDB00D447992FCCB4219B72337: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-local-dir/output/attempt_1463536071549_0001_1_00_000000_0_10029_0/file.out
2016-05-18 01:47:51,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:51 Completed running task attempt: attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,838 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:51,839 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536071549_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:51,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536071549_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536071823, Process_Duration=7, Process_End_Time=1463536071830, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:51,840 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536071549_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,840 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,840 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:51,840 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,841 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536071549_0001_00_000001
2016-05-18 01:47:51,841 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, taskAttemptId=attempt_1463536071549_0001_1_00_000000_0, creationTime=1463536071771, allocationTime=1463536071781, startTime=1463536071783, finishTime=1463536071841, timeTaken=58, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:51,842 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536071549_0001_1_00_000000_0
2016-05-18 01:47:51,843 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_FINISHED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, taskId=task_1463536071549_0001_1_00_000000, startTime=1463536071783, finishTime=1463536071842, timeTaken=59, status=SUCCEEDED, successfulAttemptID=attempt_1463536071549_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536071823, Process_Duration=7, Process_End_Time=1463536071830, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:51,843 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:51,843 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_FINISHED]: vertexName=E2763F4F362E4CC3A26F9BB12E7ED979, vertexId=vertex_1463536071549_0001_1_00, initRequestedTime=1463536071734, initedTime=1463536071761, startRequestedTime=1463536071736, startedTime=1463536071762, finishTime=1463536071843, timeTaken=81, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536071823, Process_Duration=7, Process_End_Time=1463536071830, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536071783, firstTasksToStart=[ task_1463536071549_0001_1_00_000000 ], lastTaskFinishTime=1463536071841, lastTasksToFinish=[ task_1463536071549_0001_1_00_000000 ], minTaskDuration=58, maxTaskDuration=58, avgTaskDuration=58.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536071549_0001_1_00_000000 ], longestDurationTasks=[ task_1463536071549_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:51,845 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:51,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536071549_0001_00_000001, stoppedTime=1463536071845, exitStatus=0
2016-05-18 01:47:51,845 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] attempt: attempt_1463536071549_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:51,846 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:51,846 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DA3B2CDDB00D447992FCCB4219B72337
2016-05-18 01:47:51,846 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,846 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536071549_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:51,846 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E2763F4F362E4CC3A26F9BB12E7ED979, destinationVertexName=DA3B2CDDB00D447992FCCB4219B72337. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:51,847 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_STARTED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, taskId=task_1463536071549_0001_1_01_000000, scheduledTime=1463536071847, launchTime=1463536071847
2016-05-18 01:47:51,847 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536071549_0001_00_000002, launchTime=1463536071849
2016-05-18 01:47:51,850 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536071549_0001_00_000002
2016-05-18 01:47:51,850 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536071549_0001_1_01_000000_0] started. Is using containerId: [container_1463536071549_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:51,851 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0, startTime=1463536071850, containerId=container_1463536071549_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:51,853 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536071549_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:51,853 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536071549_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,853 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536071549_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:51 Starting to run new task attempt: attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,855 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536071549_0001_00_000001, Exiting
2016-05-18 01:47:51,855 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536071549_0001_00_000001
2016-05-18 01:47:51,855 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536071549_0001_00_000001
2016-05-18 01:47:51,854 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,855 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,855 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:51,856 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false, VertexName: DA3B2CDDB00D447992FCCB4219B72337, VertexParallelism: 1, TaskAttemptID:attempt_1463536071549_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E2763F4F362E4CC3A26F9BB12E7ED979, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=330A7B22BD2D45A187E9FA8FAD740BCC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:51,856 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:51,856 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536071549_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:51,857 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,857 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536071549_0001_00_000001 completed successfully
2016-05-18 01:47:51,860 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536071549_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,861 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536071549_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:51,861 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:51,863 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:51,875 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DA3B2CDDB00D447992FCCB4219B72337, ordinal: 1
2016-05-18 01:47:51,875 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DA3B2CDDB00D447992FCCB4219B72337, mem on start (mb), free: 198, total: 355, max: 672
2016-05-18 01:47:51,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:51,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:51,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:51,878 INFO output.MROutput (MROutput.java:initialize(421)) - 330A7B22BD2D45A187E9FA8FAD740BCC: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:51,878 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:51,878 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E2763F4F362E4CC3A26F9BB12E7ED979:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [330A7B22BD2D45A187E9FA8FAD740BCC:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:51,879 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:51,879 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:51,880 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:51,880 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E2763F4F362E4CC3A26F9BB12E7ED979:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [330A7B22BD2D45A187E9FA8FAD740BCC:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:51,880 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:51,880 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E2763F4F362E4CC3A26F9BB12E7ED979 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:51,881 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E2763F4F362E4CC3A26F9BB12E7ED979: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:51,881 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E2763F4F362E4CC3A26F9BB12E7ED979: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:51,882 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E2763F4F362E4CC3A26F9BB12E7ED979 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:51,882 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:51,882 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:51,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E2763F4F362E4CC3A26F9BB12E7ED979
2016-05-18 01:47:51,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:51,884 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,886 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E2763F4F362E4CC3A26F9BB12E7ED979: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:51,888 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536071549_0001_1_00_000000_0_10029} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536071888, TimeTaken=2, Rate=0.28 MB/s
2016-05-18 01:47:51,888 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E2763F4F362E4CC3A26F9BB12E7ED979
2016-05-18 01:47:51,888 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:51,888 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:51,889 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E2763F4F362E4CC3A26F9BB12E7ED979, wasInterrupted=false
2016-05-18 01:47:51,890 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:51,892 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 0EDFF2C128EB4B5F83CE057625A8BE39
2016-05-18 01:47:51,893 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false"], id: 330A7B22BD2D45A187E9FA8FAD740BCC
2016-05-18 01:47:51,893 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:51,893 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false"]
2016-05-18 01:47:51,893 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 0EDFF2C128EB4B5F83CE057625A8BE39, for 1 inputs
2016-05-18 01:47:51,894 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001_wd/localmode-local-dir/output/attempt_1463536071549_0001_1_00_000000_0_10029/file.out, len=593, isLocal=true
2016-05-18 01:47:51,894 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:51,895 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:51,895 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:51,895 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:51,895 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E2763F4F362E4CC3A26F9BB12E7ED979
2016-05-18 01:47:51,895 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E2763F4F362E4CC3A26F9BB12E7ED979: Shuffle Runner thread complete
2016-05-18 01:47:51,896 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DA3B2CDDB00D447992FCCB4219B72337, all 1 inputs ready in: 00:00:00.003
2016-05-18 01:47:51,897 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E2763F4F362E4CC3A26F9BB12E7ED979: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:51,902 INFO output.MROutput (MROutput.java:close(526)) - 330A7B22BD2D45A187E9FA8FAD740BCC closed
2016-05-18 01:47:51,903 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536071549_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:51,904 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,904 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DA3B2CDDB00D447992FCCB4219B72337, mem on close (mb), free: 194, total: 355, max: 672
2016-05-18 01:47:51,905 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E2763F4F362E4CC3A26F9BB12E7ED979
2016-05-18 01:47:51,905 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E2763F4F362E4CC3A26F9BB12E7ED979: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:51,905 INFO output.MROutput (MROutput.java:close(526)) - 330A7B22BD2D45A187E9FA8FAD740BCC closed
2016-05-18 01:47:51,905 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:51 Completed running task attempt: attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,906 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:51,906 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536071549_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:51,906 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536071549_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536071896, Process_Duration=8, Process_End_Time=1463536071904, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:51,908 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536071549_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,908 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:51,908 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:51,908 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,908 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536071549_0001_00_000002
2016-05-18 01:47:51,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, taskAttemptId=attempt_1463536071549_0001_1_01_000000_0, creationTime=1463536071847, allocationTime=1463536071848, startTime=1463536071850, finishTime=1463536071908, timeTaken=58, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:51,909 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536071549_0001_1_01_000000_0
2016-05-18 01:47:51,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:TASK_FINISHED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, taskId=task_1463536071549_0001_1_01_000000, startTime=1463536071850, finishTime=1463536071909, timeTaken=59, status=SUCCEEDED, successfulAttemptID=attempt_1463536071549_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536071896, Process_Duration=8, Process_End_Time=1463536071904, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:51,910 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:51,910 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337]
2016-05-18 01:47:51,911 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:VERTEX_FINISHED]: vertexName=DA3B2CDDB00D447992FCCB4219B72337, vertexId=vertex_1463536071549_0001_1_01, initRequestedTime=1463536071736, initedTime=1463536071744, startRequestedTime=1463536071762, startedTime=1463536071763, finishTime=1463536071910, timeTaken=147, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536071896, Process_Duration=8, Process_End_Time=1463536071904, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536071850, firstTasksToStart=[ task_1463536071549_0001_1_01_000000 ], lastTaskFinishTime=1463536071908, lastTasksToFinish=[ task_1463536071549_0001_1_01_000000 ], minTaskDuration=58, maxTaskDuration=58, avgTaskDuration=58.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536071549_0001_1_01_000000 ], longestDurationTasks=[ task_1463536071549_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:51,912 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:51,912 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536071549_0001_00_000002, stoppedTime=1463536071912, exitStatus=0
2016-05-18 01:47:51,912 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:51,912 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,912 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536071549_0001_1_00 [E2763F4F362E4CC3A26F9BB12E7ED979]
2016-05-18 01:47:51,912 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536071549_0001_1
2016-05-18 01:47:51,913 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536071549_0001_1
2016-05-18 01:47:51,913 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 330A7B22BD2D45A187E9FA8FAD740BCC for vertex: vertex_1463536071549_0001_1_01 [DA3B2CDDB00D447992FCCB4219B72337], outputName: 330A7B22BD2D45A187E9FA8FAD740BCC
2016-05-18 01:47:51,913 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536071549_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:51,913 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536071549_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:51,919 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536071549_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:51,919 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536071549_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:51,919 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536071549_0001_00_000002, Exiting
2016-05-18 01:47:51,919 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536071549_0001_00_000002
2016-05-18 01:47:51,919 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536071549_0001_00_000002
2016-05-18 01:47:51,919 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536071549_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:51,920 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536071549_0001_00_000002 completed successfully
2016-05-18 01:47:51,924 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:330A7B22BD2D45A187E9FA8FAD740BCC of vertex/vertexGroup:DA3B2CDDB00D447992FCCB4219B72337 isVertexGroupOutput:false
2016-05-18 01:47:51,924 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:51,926 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536071549_0001_1, queueSize=0
2016-05-18 01:47:51,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536071549_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536071549_0001_1, startTime=1463536071733, finishTime=1463536071924, timeTaken=191, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072143719, Process_Duration=15, Process_End_Time=2927072143734, Read_Duration=3, Tuples_Read=74, Tuples_Written=74, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:51,927 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536071549_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:51,927 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536071549_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:51 Completed Dag: dag_1463536071549_0001_1
2016-05-18 01:47:51,928 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536071549_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:51,928 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536071549_0001_1_post
2016-05-18 01:47:51,928 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:51,928 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:51,929 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:51,929 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false, with id=dag_1463536071549_0001_1
2016-05-18 01:47:51,929 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492] comprehensivesort/(1/1) ...ive/true_null_false_false, with id=dag_1463536071549_0001_1
2016-05-18 01:47:51,936 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/true_null_false_false, applicationId=application_1463536071549_0001
2016-05-18 01:47:51,940 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:51,940 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:51,940 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:51,940 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:51,940 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:51,941 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:51,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:51,942 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:51,942 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:51,942 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:51,943 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:51,943 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:51,943 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:51,945 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:51,948 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/99F0472CE790454DB9E904C688827EE6/AEFAF4EC43F1463F934BE4D10846B492/.tez/application_1463536071549_0001
2016-05-18 01:47:51,949 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false/_temporary
2016-05-18 01:47:51,949 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.422
2016-05-18 01:47:52,033 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:52,033 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:52,083 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 01:47:52,090 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.027
2016-05-18 01:47:52,091 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:52,091 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:52,092 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:52,110 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:52,119 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false
2016-05-18 01:47:52,176 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:52,176 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:52,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false"]
2016-05-18 01:47:52,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:52,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:52,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:52,178 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/false_null_true_false
2016-05-18 01:47:52,178 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:52,193 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:52,197 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:52,198 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:52,198 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:52,208 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001 doesn't exist and is created
2016-05-18 01:47:52,229 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:52,229 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:52,231 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd
2016-05-18 01:47:52,247 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536072198_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:52,247 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:52,247 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:52,249 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:52,249 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:52,249 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:52,249 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:52,250 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:52,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:52,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:52,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:52,252 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:52,252 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:52,253 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:52,253 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:52,254 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:52,255 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:52,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:52,255 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-log-dir
2016-05-18 01:47:52,256 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-log-dir/history.txt.appattempt_1463536072198_0001_000000, maxErrors=10
2016-05-18 01:47:52,256 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:52,256 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536072198_0001_000000, appSubmitTime=1463536072246, launchTime=1463536072246
2016-05-18 01:47:52,261 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:52,273 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:45610
2016-05-18 01:47:52,275 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536072198
2016-05-18 01:47:52,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536072198_0001_000000, startTime=1463536072276
2016-05-18 01:47:52,277 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:52,330 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:52,330 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:52,330 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/false_null_true_false, applicationId=application_1463536072198_0001, dagName=[3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false
2016-05-18 01:47:52,350 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false
2016-05-18 01:47:52,350 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:52,351 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536072198_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-log-dir/dag_1463536072198_0001_1.dot
2016-05-18 01:47:52,352 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-log-dir/dag_1463536072198_0001_1-tez-dag.pb.txt
2016-05-18 01:47:52,363 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536072198_0001_1
2016-05-18 01:47:52,363 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,363 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false
2016-05-18 01:47:52 Running Dag: dag_1463536072198_0001_1
2016-05-18 01:47:52,380 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536072198_0001_1, submitTime=1463536072350
2016-05-18 01:47:52,381 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001/tez-conf.pb" } size: 40937 timestamp: 1463536072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:52,381 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,382 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:52,382 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001/tez-conf.pb" } size: 40937 timestamp: 1463536072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:52,383 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,383 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:52,383 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:52,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536072198_0001_1, initTime=1463536072384
2016-05-18 01:47:52,385 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072198_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:52,385 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:52,386 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:DAG_STARTED]: dagID=dag_1463536072198_0001_1, startTime=1463536072385
2016-05-18 01:47:52,386 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072198_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:52,386 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/false_null_true_false, applicationId=application_1463536072198_0001, dagName=[3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false
2016-05-18 01:47:52,386 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AF59C9388A33431A910EAE917D94C6CC : {A8A9CBBA89BA441C84ADA35F072416A7={InputName=A8A9CBBA89BA441C84ADA35F072416A7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:52,386 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A8A9CBBA89BA441C84ADA35F072416A7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:52,386 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,386 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,386 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,387 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,387 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:52,388 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,391 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:52,391 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,392 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,392 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536072198_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:52,392 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB], numAdditionalOutputs=1
2016-05-18 01:47:52,394 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=52AFF23BE3C041508281F1D94B59B0D0, vertex=vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:52,387 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A8A9CBBA89BA441C84ADA35F072416A7 on vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,399 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CC818347708248CFBF00EFA7DB87CCDB:52AFF23BE3C041508281F1D94B59B0D0 using oldmapred API
2016-05-18 01:47:52,400 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=52AFF23BE3C041508281F1D94B59B0D0, vertexName=CC818347708248CFBF00EFA7DB87CCDB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:52,400 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A8A9CBBA89BA441C84ADA35F072416A7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:52,401 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, vertexId=vertex_1463536072198_0001_1_01, initRequestedTime=1463536072388, initedTime=1463536072392, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:52,401 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:52,403 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:52,403 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:52,403 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:52,404 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:52,404 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A8A9CBBA89BA441C84ADA35F072416A7 on vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,406 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:52,406 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A8A9CBBA89BA441C84ADA35F072416A7=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:52,407 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] parallelism set to 1
2016-05-18 01:47:52,407 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:52,407 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, vertexId=vertex_1463536072198_0001_1_00, initRequestedTime=1463536072386, initedTime=1463536072407, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:52,407 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] with distanceFromRoot: 0
2016-05-18 01:47:52,408 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:52,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536072198_0001_1_00, startRequestedTime=1463536072388, startedTime=1463536072408
2016-05-18 01:47:52,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536072198_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:52,408 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:52,408 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536072198_0001_1_00 for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] numStartedSources: 1 numSources: 1
2016-05-18 01:47:52,409 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] with distanceFromRoot: 1
2016-05-18 01:47:52,409 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536072198_0001_1_01, startRequestedTime=1463536072408, startedTime=1463536072409
2016-05-18 01:47:52,409 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:52,413 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CC818347708248CFBF00EFA7DB87CCDB with 0 source tasks and 1 pending tasks
2016-05-18 01:47:52,413 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AF59C9388A33431A910EAE917D94C6CC
2016-05-18 01:47:52,414 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,414 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_STARTED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, taskId=task_1463536072198_0001_1_00_000000, scheduledTime=1463536072414, launchTime=1463536072414
2016-05-18 01:47:52,414 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AF59C9388A33431A910EAE917D94C6CC in vertex: CC818347708248CFBF00EFA7DB87CCDB numBipartiteSourceTasks: 1
2016-05-18 01:47:52,415 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,416 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:52,417 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536072198_0001_00_000001, launchTime=1463536072417
2016-05-18 01:47:52,418 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072198_0001_00_000001
2016-05-18 01:47:52,419 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072198_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,419 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536072198_0001_1_00_000000_0] started. Is using containerId: [container_1463536072198_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:52,419 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0, startTime=1463536072419, containerId=container_1463536072198_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:52 Starting to run new task attempt: attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,419 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,419 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,419 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:52,420 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false, VertexName: AF59C9388A33431A910EAE917D94C6CC, VertexParallelism: 1, TaskAttemptID:attempt_1463536072198_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A8A9CBBA89BA441C84ADA35F072416A7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CC818347708248CFBF00EFA7DB87CCDB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:52,420 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:52,420 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,421 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536072198_0001, with dag id: dag_1463536072198_0001_1
2016-05-18 01:47:52,423 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536072198_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:52,423 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536072198_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:52,425 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:52,434 INFO input.MRInput (MRInput.java:initialize(440)) - A8A9CBBA89BA441C84ADA35F072416A7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:52,434 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A8A9CBBA89BA441C84ADA35F072416A7
2016-05-18 01:47:52,438 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AF59C9388A33431A910EAE917D94C6CC, ordinal: 0
2016-05-18 01:47:52,438 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF59C9388A33431A910EAE917D94C6CC, mem on start (mb), free: 305, total: 356, max: 672
2016-05-18 01:47:52,438 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:52,438 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:52,438 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:52,438 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:52,438 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CC818347708248CFBF00EFA7DB87CCDB:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [A8A9CBBA89BA441C84ADA35F072416A7:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 01:47:52,439 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:52,439 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:52,439 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:52,439 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CC818347708248CFBF00EFA7DB87CCDB:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [A8A9CBBA89BA441C84ADA35F072416A7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 01:47:52,439 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:52,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A8A9CBBA89BA441C84ADA35F072416A7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:52,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:52,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:52,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A8A9CBBA89BA441C84ADA35F072416A7
2016-05-18 01:47:52,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:52,441 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,445 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:52,445 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A8A9CBBA89BA441C84ADA35F072416A7 initialized RecordReader from event
2016-05-18 01:47:52,448 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:52,449 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: A8A9CBBA89BA441C84ADA35F072416A7
2016-05-18 01:47:52,449 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: B5C772AFB76544BEB0E831A93C1D5291
2016-05-18 01:47:52,449 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] B5C772AFB76544BEB0E831A93C1D5291
2016-05-18 01:47:52,452 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CC818347708248CFBF00EFA7DB87CCDB using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@cb1b79, comparator=cascading.tuple.hadoop.util.ReverseTupleComparator@1678a557, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:52,452 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CC818347708248CFBF00EFA7DB87CCDB: , UsingHashComparator=false
2016-05-18 01:47:52,468 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:52,468 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:52,468 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CC818347708248CFBF00EFA7DB87CCDB: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:52,469 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CC818347708248CFBF00EFA7DB87CCDB: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:52,469 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:52,470 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AF59C9388A33431A910EAE917D94C6CC, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:52,478 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,479 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF59C9388A33431A910EAE917D94C6CC, mem on close (mb), free: 203, total: 356, max: 672
2016-05-18 01:47:52,479 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CC818347708248CFBF00EFA7DB87CCDB: Starting flush of map output
2016-05-18 01:47:52,479 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CC818347708248CFBF00EFA7DB87CCDB: Span0.length = 37, perItem = 17
2016-05-18 01:47:52,480 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CC818347708248CFBF00EFA7DB87CCDB: done sorting span=0, length=37, time=0
2016-05-18 01:47:52,481 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CC818347708248CFBF00EFA7DB87CCDB: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:52,482 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CC818347708248CFBF00EFA7DB87CCDB: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-local-dir/output/attempt_1463536072198_0001_1_00_000000_0_10034_0/file.out
2016-05-18 01:47:52,486 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:52 Completed running task attempt: attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,487 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:52,487 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536072198_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:52,488 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536072198_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536072471, Process_Duration=7, Process_End_Time=1463536072478, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:52,489 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536072198_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:52,489 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:52,489 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:52,489 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,490 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, taskAttemptId=attempt_1463536072198_0001_1_00_000000_0, creationTime=1463536072415, allocationTime=1463536072416, startTime=1463536072419, finishTime=1463536072489, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:52,490 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536072198_0001_1_00_000000_0
2016-05-18 01:47:52,490 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072198_0001_00_000001
2016-05-18 01:47:52,491 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_FINISHED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, taskId=task_1463536072198_0001_1_00_000000, startTime=1463536072419, finishTime=1463536072490, timeTaken=71, status=SUCCEEDED, successfulAttemptID=attempt_1463536072198_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536072471, Process_Duration=7, Process_End_Time=1463536072478, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:52,491 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:52,492 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,493 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_FINISHED]: vertexName=AF59C9388A33431A910EAE917D94C6CC, vertexId=vertex_1463536072198_0001_1_00, initRequestedTime=1463536072386, initedTime=1463536072407, startRequestedTime=1463536072388, startedTime=1463536072408, finishTime=1463536072492, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536072471, Process_Duration=7, Process_End_Time=1463536072478, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536072419, firstTasksToStart=[ task_1463536072198_0001_1_00_000000 ], lastTaskFinishTime=1463536072489, lastTasksToFinish=[ task_1463536072198_0001_1_00_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536072198_0001_1_00_000000 ], longestDurationTasks=[ task_1463536072198_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:52,493 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:52,493 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536072198_0001_00_000001, stoppedTime=1463536072493, exitStatus=0
2016-05-18 01:47:52,493 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] attempt: attempt_1463536072198_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:52,494 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:52,494 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CC818347708248CFBF00EFA7DB87CCDB
2016-05-18 01:47:52,494 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:52,494 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AF59C9388A33431A910EAE917D94C6CC, destinationVertexName=CC818347708248CFBF00EFA7DB87CCDB. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:52,494 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536072198_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:52,494 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,495 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_STARTED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, taskId=task_1463536072198_0001_1_01_000000, scheduledTime=1463536072495, launchTime=1463536072495
2016-05-18 01:47:52,495 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536072198_0001_00_000002, launchTime=1463536072497
2016-05-18 01:47:52,497 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072198_0001_00_000002
2016-05-18 01:47:52,499 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536072198_0001_1_01_000000_0] started. Is using containerId: [container_1463536072198_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:52,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0, startTime=1463536072499, containerId=container_1463536072198_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:52,500 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072198_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,501 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536072198_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:52,501 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072198_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:52,502 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536072198_0001_00_000001, Exiting
2016-05-18 01:47:52,502 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536072198_0001_00_000001
2016-05-18 01:47:52,502 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536072198_0001_00_000001
2016-05-18 01:47:52 Starting to run new task attempt: attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,503 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536072198_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:52,503 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,503 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,503 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536072198_0001_00_000001 completed successfully
2016-05-18 01:47:52,504 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:52,504 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false, VertexName: CC818347708248CFBF00EFA7DB87CCDB, VertexParallelism: 1, TaskAttemptID:attempt_1463536072198_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AF59C9388A33431A910EAE917D94C6CC, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=52AFF23BE3C041508281F1D94B59B0D0, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:52,504 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:52,505 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,508 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:52,511 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:52,513 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536072198_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,513 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536072198_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:52,520 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CC818347708248CFBF00EFA7DB87CCDB, ordinal: 1
2016-05-18 01:47:52,520 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC818347708248CFBF00EFA7DB87CCDB, mem on start (mb), free: 197, total: 356, max: 672
2016-05-18 01:47:52,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:52,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:52,521 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:52,523 INFO output.MROutput (MROutput.java:initialize(421)) - 52AFF23BE3C041508281F1D94B59B0D0: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:52,523 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:52,523 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[52AFF23BE3C041508281F1D94B59B0D0:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [AF59C9388A33431A910EAE917D94C6CC:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:52,523 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:52,523 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:52,524 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:52,524 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[52AFF23BE3C041508281F1D94B59B0D0:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [AF59C9388A33431A910EAE917D94C6CC:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:52,524 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:52,525 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AF59C9388A33431A910EAE917D94C6CC: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:52,525 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AF59C9388A33431A910EAE917D94C6CC: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:52,528 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AF59C9388A33431A910EAE917D94C6CC with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:52,528 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AF59C9388A33431A910EAE917D94C6CC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:52,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:52,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:52,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AF59C9388A33431A910EAE917D94C6CC
2016-05-18 01:47:52,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:52,536 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,537 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AF59C9388A33431A910EAE917D94C6CC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:52,540 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:52,541 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: B5C772AFB76544BEB0E831A93C1D5291
2016-05-18 01:47:52,541 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false"], id: 52AFF23BE3C041508281F1D94B59B0D0
2016-05-18 01:47:52,541 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536072198_0001_1_00_000000_0_10034} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536072541, TimeTaken=1, Rate=0.57 MB/s
2016-05-18 01:47:52,541 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false"]
2016-05-18 01:47:52,542 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AF59C9388A33431A910EAE917D94C6CC
2016-05-18 01:47:52,542 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] B5C772AFB76544BEB0E831A93C1D5291, for 1 inputs
2016-05-18 01:47:52,542 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:52,544 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AF59C9388A33431A910EAE917D94C6CC, wasInterrupted=false
2016-05-18 01:47:52,544 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:52,545 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:52,546 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001_wd/localmode-local-dir/output/attempt_1463536072198_0001_1_00_000000_0_10034/file.out, len=593, isLocal=true
2016-05-18 01:47:52,546 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:52,546 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:52,546 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:52,547 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:52,547 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AF59C9388A33431A910EAE917D94C6CC
2016-05-18 01:47:52,547 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AF59C9388A33431A910EAE917D94C6CC: Shuffle Runner thread complete
2016-05-18 01:47:52,547 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CC818347708248CFBF00EFA7DB87CCDB, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:47:52,548 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AF59C9388A33431A910EAE917D94C6CC: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseTupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:52,554 INFO output.MROutput (MROutput.java:close(526)) - 52AFF23BE3C041508281F1D94B59B0D0 closed
2016-05-18 01:47:52,555 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536072198_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:52,556 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,556 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC818347708248CFBF00EFA7DB87CCDB, mem on close (mb), free: 194, total: 356, max: 672
2016-05-18 01:47:52,556 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AF59C9388A33431A910EAE917D94C6CC
2016-05-18 01:47:52,556 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AF59C9388A33431A910EAE917D94C6CC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:52,557 INFO output.MROutput (MROutput.java:close(526)) - 52AFF23BE3C041508281F1D94B59B0D0 closed
2016-05-18 01:47:52,557 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:52 Completed running task attempt: attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,557 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:52,557 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536072198_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:52,558 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536072198_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=19, MERGE_PHASE_TIME=22, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536072547, Process_Duration=9, Process_End_Time=1463536072556, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:52,560 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536072198_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:52,560 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, taskAttemptId=attempt_1463536072198_0001_1_01_000000_0, creationTime=1463536072495, allocationTime=1463536072496, startTime=1463536072499, finishTime=1463536072560, timeTaken=61, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:52,560 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:52,560 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:52,560 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536072198_0001_1_01_000000_0
2016-05-18 01:47:52,561 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,561 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072198_0001_00_000002
2016-05-18 01:47:52,561 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:TASK_FINISHED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, taskId=task_1463536072198_0001_1_01_000000, startTime=1463536072499, finishTime=1463536072561, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1463536072198_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=19, MERGE_PHASE_TIME=22, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536072547, Process_Duration=9, Process_End_Time=1463536072556, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:52,561 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:52,562 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB]
2016-05-18 01:47:52,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:VERTEX_FINISHED]: vertexName=CC818347708248CFBF00EFA7DB87CCDB, vertexId=vertex_1463536072198_0001_1_01, initRequestedTime=1463536072388, initedTime=1463536072392, startRequestedTime=1463536072408, startedTime=1463536072409, finishTime=1463536072562, timeTaken=153, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=19, MERGE_PHASE_TIME=22, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536072547, Process_Duration=9, Process_End_Time=1463536072556, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536072499, firstTasksToStart=[ task_1463536072198_0001_1_01_000000 ], lastTaskFinishTime=1463536072560, lastTasksToFinish=[ task_1463536072198_0001_1_01_000000 ], minTaskDuration=61, maxTaskDuration=61, avgTaskDuration=61.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536072198_0001_1_01_000000 ], longestDurationTasks=[ task_1463536072198_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:52,563 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:52,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536072198_0001_00_000002, stoppedTime=1463536072563, exitStatus=0
2016-05-18 01:47:52,563 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:52,563 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:52,563 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536072198_0001_1_00 [AF59C9388A33431A910EAE917D94C6CC]
2016-05-18 01:47:52,563 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536072198_0001_1
2016-05-18 01:47:52,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536072198_0001_1
2016-05-18 01:47:52,564 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 52AFF23BE3C041508281F1D94B59B0D0 for vertex: vertex_1463536072198_0001_1_01 [CC818347708248CFBF00EFA7DB87CCDB], outputName: 52AFF23BE3C041508281F1D94B59B0D0
2016-05-18 01:47:52,567 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072198_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:52,567 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536072198_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:52,571 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536072198_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:52,571 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072198_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:52,571 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536072198_0001_00_000002, Exiting
2016-05-18 01:47:52,572 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536072198_0001_00_000002
2016-05-18 01:47:52,572 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536072198_0001_00_000002
2016-05-18 01:47:52,572 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536072198_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:52,572 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536072198_0001_00_000002 completed successfully
2016-05-18 01:47:52,574 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:52AFF23BE3C041508281F1D94B59B0D0 of vertex/vertexGroup:CC818347708248CFBF00EFA7DB87CCDB isVertexGroupOutput:false
2016-05-18 01:47:52,574 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:52,576 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536072198_0001_1, queueSize=0
2016-05-18 01:47:52,577 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072198_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536072198_0001_1, startTime=1463536072385, finishTime=1463536072574, timeTaken=189, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=370, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=19, MERGE_PHASE_TIME=22, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072145018, Process_Duration=16, Process_End_Time=2927072145034, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:52,577 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536072198_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:52,577 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072198_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:52 Completed Dag: dag_1463536072198_0001_1
2016-05-18 01:47:52,577 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536072198_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:52,577 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536072198_0001_1_post
2016-05-18 01:47:52,578 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:52,578 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:52,578 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:52,578 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false, with id=dag_1463536072198_0001_1
2016-05-18 01:47:52,578 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB] comprehensivesort/(1/1) ...ive/false_null_true_false, with id=dag_1463536072198_0001_1
2016-05-18 01:47:52,586 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/false_null_true_false, applicationId=application_1463536072198_0001
2016-05-18 01:47:52,588 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:52,588 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:52,588 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:52,588 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:52,589 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:52,589 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:52,589 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:52,590 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:52,590 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:52,590 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:52,591 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:52,592 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:52,592 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:52,593 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:52,597 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3C1898C41B7248B68B000E66C22EE45E/F60B882FBD784A86A2688B4FB36E56BB/.tez/application_1463536072198_0001
2016-05-18 01:47:52,598 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false/_temporary
2016-05-18 01:47:52,599 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.422
2016-05-18 01:47:52,678 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:52,678 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:52,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.021
2016-05-18 01:47:52,748 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.039
2016-05-18 01:47:52,748 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:52,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:52,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:52,769 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:52,778 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false
2016-05-18 01:47:52,840 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:52,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:52,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false"]
2016-05-18 01:47:52,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:52,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:52,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:52,842 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/true_null_true_false
2016-05-18 01:47:52,843 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:52,858 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:52,862 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:52,863 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:52,863 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:52,873 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001 doesn't exist and is created
2016-05-18 01:47:52,895 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:52,896 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:52,897 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd
2016-05-18 01:47:52,913 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536072863_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:52,913 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:52,914 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:52,916 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:52,916 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:52,916 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:52,916 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:52,918 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:52,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:52,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:52,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:52,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:52,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:52,921 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:52,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:52,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:52,923 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:52,923 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:52,924 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:52,924 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:52,926 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:52,927 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:52,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:52,927 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-log-dir
2016-05-18 01:47:52,928 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-log-dir/history.txt.appattempt_1463536072863_0001_000000, maxErrors=10
2016-05-18 01:47:52,928 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:52,929 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536072863_0001_000000, appSubmitTime=1463536072912, launchTime=1463536072912
2016-05-18 01:47:52,937 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:52,946 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:57693
2016-05-18 01:47:52,955 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536072863
2016-05-18 01:47:52,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536072863_0001_000000, startTime=1463536072957
2016-05-18 01:47:52,957 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:52,996 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:52,996 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:52,996 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/true_null_true_false, applicationId=application_1463536072863_0001, dagName=[5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false
2016-05-18 01:47:53,017 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false
2016-05-18 01:47:53,017 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:53,018 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536072863_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-log-dir/dag_1463536072863_0001_1.dot
2016-05-18 01:47:53,019 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-log-dir/dag_1463536072863_0001_1-tez-dag.pb.txt
2016-05-18 01:47:53,029 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536072863_0001_1
2016-05-18 01:47:53,029 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,029 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false
2016-05-18 01:47:53 Running Dag: dag_1463536072863_0001_1
2016-05-18 01:47:53,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536072863_0001_1, submitTime=1463536073017
2016-05-18 01:47:53,046 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001/tez-conf.pb" } size: 40937 timestamp: 1463536072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:53,046 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,046 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:53,047 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001/tez-conf.pb" } size: 40937 timestamp: 1463536072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:53,047 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,048 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:53,048 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:53,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536072863_0001_1, initTime=1463536073048
2016-05-18 01:47:53,049 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072863_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:53,050 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:53,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:DAG_STARTED]: dagID=dag_1463536072863_0001_1, startTime=1463536073050
2016-05-18 01:47:53,050 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072863_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:53,050 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B2F07668EF9048C280EEE3C510CF4612 : {3482097F90544D0E961C967DC0920492={InputName=3482097F90544D0E961C967DC0920492}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:53,050 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 3482097F90544D0E961C967DC0920492, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:53,050 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,051 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,051 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,052 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,053 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/true_null_true_false, applicationId=application_1463536072863_0001, dagName=[5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false
2016-05-18 01:47:53,054 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:53,054 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,054 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 3482097F90544D0E961C967DC0920492 on vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,058 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:53,058 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,058 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536072863_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:53,062 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58], numAdditionalOutputs=1
2016-05-18 01:47:53,062 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=5CB09D6B115F4F799DB43E3C7AFDDA39, vertex=vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:53,063 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 3482097F90544D0E961C967DC0920492 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:53,065 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B5AF6EEFD1094908A33E4023ED163B58:5CB09D6B115F4F799DB43E3C7AFDDA39 using oldmapred API
2016-05-18 01:47:53,065 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:53,066 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:53,066 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=5CB09D6B115F4F799DB43E3C7AFDDA39, vertexName=B5AF6EEFD1094908A33E4023ED163B58, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:53,066 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:53,067 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:53,068 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 3482097F90544D0E961C967DC0920492 on vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, vertexId=vertex_1463536072863_0001_1_01, initRequestedTime=1463536073054, initedTime=1463536073062, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:53,068 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:53,069 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:53,069 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {3482097F90544D0E961C967DC0920492=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:53,070 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] parallelism set to 1
2016-05-18 01:47:53,071 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:53,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, vertexId=vertex_1463536072863_0001_1_00, initRequestedTime=1463536073050, initedTime=1463536073071, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:53,072 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] with distanceFromRoot: 0
2016-05-18 01:47:53,072 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:53,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536072863_0001_1_00, startRequestedTime=1463536073054, startedTime=1463536073072
2016-05-18 01:47:53,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536072863_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:53,073 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:53,074 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536072863_0001_1_00 for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] numStartedSources: 1 numSources: 1
2016-05-18 01:47:53,074 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] with distanceFromRoot: 1
2016-05-18 01:47:53,074 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B2F07668EF9048C280EEE3C510CF4612
2016-05-18 01:47:53,075 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,075 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536072863_0001_1_01, startRequestedTime=1463536073074, startedTime=1463536073074
2016-05-18 01:47:53,076 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:53,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_STARTED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, taskId=task_1463536072863_0001_1_00_000000, scheduledTime=1463536073078, launchTime=1463536073078
2016-05-18 01:47:53,078 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B5AF6EEFD1094908A33E4023ED163B58 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:53,079 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,079 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B2F07668EF9048C280EEE3C510CF4612 in vertex: B5AF6EEFD1094908A33E4023ED163B58 numBipartiteSourceTasks: 1
2016-05-18 01:47:53,080 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:53,082 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536072863_0001_00_000001, launchTime=1463536073082
2016-05-18 01:47:53,082 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072863_0001_00_000001
2016-05-18 01:47:53,084 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072863_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,084 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536072863_0001_1_00_000000_0] started. Is using containerId: [container_1463536072863_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:53,084 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0, startTime=1463536073084, containerId=container_1463536072863_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:53 Starting to run new task attempt: attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,085 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,085 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,085 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:53,086 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false, VertexName: B2F07668EF9048C280EEE3C510CF4612, VertexParallelism: 1, TaskAttemptID:attempt_1463536072863_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3482097F90544D0E961C967DC0920492, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B5AF6EEFD1094908A33E4023ED163B58, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:53,086 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:53,086 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,087 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536072863_0001, with dag id: dag_1463536072863_0001_1
2016-05-18 01:47:53,087 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536072863_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:53,088 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536072863_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:53,092 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:53,094 INFO input.MRInput (MRInput.java:initialize(440)) - 3482097F90544D0E961C967DC0920492 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:53,094 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 3482097F90544D0E961C967DC0920492
2016-05-18 01:47:53,100 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B2F07668EF9048C280EEE3C510CF4612, ordinal: 0
2016-05-18 01:47:53,100 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2F07668EF9048C280EEE3C510CF4612, mem on start (mb), free: 292, total: 345, max: 672
2016-05-18 01:47:53,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:53,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:53,101 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:53,101 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:53,101 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[3482097F90544D0E961C967DC0920492:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B5AF6EEFD1094908A33E4023ED163B58:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:53,101 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:53,101 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:53,101 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:53,102 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[3482097F90544D0E961C967DC0920492:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B5AF6EEFD1094908A33E4023ED163B58:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:53,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:53,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 3482097F90544D0E961C967DC0920492 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:53,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:53,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 3482097F90544D0E961C967DC0920492
2016-05-18 01:47:53,102 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:53,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:53,103 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,106 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:53,106 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 3482097F90544D0E961C967DC0920492 initialized RecordReader from event
2016-05-18 01:47:53,107 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:53,108 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 3482097F90544D0E961C967DC0920492
2016-05-18 01:47:53,108 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: A7C20488FD904075B07071D96FCF9651
2016-05-18 01:47:53,108 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A7C20488FD904075B07071D96FCF9651
2016-05-18 01:47:53,111 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B5AF6EEFD1094908A33E4023ED163B58 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@250b0af0, comparator=cascading.tuple.hadoop.util.ReverseTupleComparator@77fb258f, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:53,112 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B5AF6EEFD1094908A33E4023ED163B58: , UsingHashComparator=false
2016-05-18 01:47:53,124 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:53,125 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:53,125 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B5AF6EEFD1094908A33E4023ED163B58: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:53,125 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B5AF6EEFD1094908A33E4023ED163B58: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:53,126 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:53,126 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B2F07668EF9048C280EEE3C510CF4612, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:53,131 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,131 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2F07668EF9048C280EEE3C510CF4612, mem on close (mb), free: 190, total: 345, max: 672
2016-05-18 01:47:53,132 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B5AF6EEFD1094908A33E4023ED163B58: Starting flush of map output
2016-05-18 01:47:53,132 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B5AF6EEFD1094908A33E4023ED163B58: Span0.length = 37, perItem = 17
2016-05-18 01:47:53,133 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B5AF6EEFD1094908A33E4023ED163B58: done sorting span=0, length=37, time=1
2016-05-18 01:47:53,133 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B5AF6EEFD1094908A33E4023ED163B58: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:53,134 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B5AF6EEFD1094908A33E4023ED163B58: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-local-dir/output/attempt_1463536072863_0001_1_00_000000_0_10040_0/file.out
2016-05-18 01:47:53,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:53 Completed running task attempt: attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,138 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:53,139 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536072863_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:53,139 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536072863_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536073126, Process_Duration=5, Process_End_Time=1463536073131, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:53,140 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536072863_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,140 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,140 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, taskAttemptId=attempt_1463536072863_0001_1_00_000000_0, creationTime=1463536073079, allocationTime=1463536073081, startTime=1463536073084, finishTime=1463536073140, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:53,141 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536072863_0001_1_00_000000_0
2016-05-18 01:47:53,140 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:53,141 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,141 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072863_0001_00_000001
2016-05-18 01:47:53,142 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_FINISHED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, taskId=task_1463536072863_0001_1_00_000000, startTime=1463536073084, finishTime=1463536073141, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463536072863_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536073126, Process_Duration=5, Process_End_Time=1463536073131, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:53,142 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:53,142 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,143 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_FINISHED]: vertexName=B2F07668EF9048C280EEE3C510CF4612, vertexId=vertex_1463536072863_0001_1_00, initRequestedTime=1463536073050, initedTime=1463536073071, startRequestedTime=1463536073054, startedTime=1463536073072, finishTime=1463536073142, timeTaken=70, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536073126, Process_Duration=5, Process_End_Time=1463536073131, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536073084, firstTasksToStart=[ task_1463536072863_0001_1_00_000000 ], lastTaskFinishTime=1463536073140, lastTasksToFinish=[ task_1463536072863_0001_1_00_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536072863_0001_1_00_000000 ], longestDurationTasks=[ task_1463536072863_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:53,144 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:53,144 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] attempt: attempt_1463536072863_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:53,144 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:53,144 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,144 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B5AF6EEFD1094908A33E4023ED163B58
2016-05-18 01:47:53,145 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536072863_0001_00_000001, stoppedTime=1463536073145, exitStatus=0
2016-05-18 01:47:53,145 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B2F07668EF9048C280EEE3C510CF4612, destinationVertexName=B5AF6EEFD1094908A33E4023ED163B58. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:53,145 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,145 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536072863_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:53,145 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072863_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:53,145 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536072863_0001_00_000001, Exiting
2016-05-18 01:47:53,145 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536072863_0001_00_000001
2016-05-18 01:47:53,146 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536072863_0001_00_000001
2016-05-18 01:47:53,146 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536072863_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:53,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_STARTED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, taskId=task_1463536072863_0001_1_01_000000, scheduledTime=1463536073146, launchTime=1463536073146
2016-05-18 01:47:53,146 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,147 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536072863_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:53,148 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536072863_0001_00_000001 completed successfully
2016-05-18 01:47:53,148 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536072863_0001_00_000002, launchTime=1463536073148
2016-05-18 01:47:53,149 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072863_0001_00_000002
2016-05-18 01:47:53,150 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536072863_0001_1_01_000000_0] started. Is using containerId: [container_1463536072863_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:53,151 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0, startTime=1463536073150, containerId=container_1463536072863_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:53,151 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072863_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53 Starting to run new task attempt: attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,152 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,152 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,152 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:53,153 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false, VertexName: B5AF6EEFD1094908A33E4023ED163B58, VertexParallelism: 1, TaskAttemptID:attempt_1463536072863_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B2F07668EF9048C280EEE3C510CF4612, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=5CB09D6B115F4F799DB43E3C7AFDDA39, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:53,153 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:53,154 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,154 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536072863_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,154 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536072863_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:53,156 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:53,158 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:53,167 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B5AF6EEFD1094908A33E4023ED163B58, ordinal: 1
2016-05-18 01:47:53,167 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B5AF6EEFD1094908A33E4023ED163B58, mem on start (mb), free: 185, total: 345, max: 672
2016-05-18 01:47:53,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:53,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:53,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:53,171 INFO output.MROutput (MROutput.java:initialize(421)) - 5CB09D6B115F4F799DB43E3C7AFDDA39: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:53,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:53,172 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B2F07668EF9048C280EEE3C510CF4612:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [5CB09D6B115F4F799DB43E3C7AFDDA39:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:53,172 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:53,172 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:53,172 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:53,172 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B2F07668EF9048C280EEE3C510CF4612:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [5CB09D6B115F4F799DB43E3C7AFDDA39:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:53,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:53,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B2F07668EF9048C280EEE3C510CF4612 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:53,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:53,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:53,173 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B2F07668EF9048C280EEE3C510CF4612: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:53,174 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B2F07668EF9048C280EEE3C510CF4612: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:53,174 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B2F07668EF9048C280EEE3C510CF4612 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:53,177 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B2F07668EF9048C280EEE3C510CF4612
2016-05-18 01:47:53,178 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:53,178 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,179 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B2F07668EF9048C280EEE3C510CF4612: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:53,182 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536072863_0001_1_00_000000_0_10040} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536073182, TimeTaken=1, Rate=0.57 MB/s
2016-05-18 01:47:53,183 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B2F07668EF9048C280EEE3C510CF4612
2016-05-18 01:47:53,183 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:53,183 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B2F07668EF9048C280EEE3C510CF4612, wasInterrupted=false
2016-05-18 01:47:53,183 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:53,184 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:53,185 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:53,185 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: A7C20488FD904075B07071D96FCF9651
2016-05-18 01:47:53,185 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false"], id: 5CB09D6B115F4F799DB43E3C7AFDDA39
2016-05-18 01:47:53,186 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001_wd/localmode-local-dir/output/attempt_1463536072863_0001_1_00_000000_0_10040/file.out, len=593, isLocal=true
2016-05-18 01:47:53,186 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:53,186 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:53,186 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:53,186 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false"]
2016-05-18 01:47:53,186 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:53,187 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B2F07668EF9048C280EEE3C510CF4612
2016-05-18 01:47:53,187 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A7C20488FD904075B07071D96FCF9651, for 1 inputs
2016-05-18 01:47:53,187 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B2F07668EF9048C280EEE3C510CF4612: Shuffle Runner thread complete
2016-05-18 01:47:53,187 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B5AF6EEFD1094908A33E4023ED163B58, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:53,188 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B2F07668EF9048C280EEE3C510CF4612: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseTupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:53,193 INFO output.MROutput (MROutput.java:close(526)) - 5CB09D6B115F4F799DB43E3C7AFDDA39 closed
2016-05-18 01:47:53,194 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536072863_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:53,195 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,195 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B5AF6EEFD1094908A33E4023ED163B58, mem on close (mb), free: 181, total: 345, max: 672
2016-05-18 01:47:53,195 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B2F07668EF9048C280EEE3C510CF4612
2016-05-18 01:47:53,195 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B2F07668EF9048C280EEE3C510CF4612: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:53,196 INFO output.MROutput (MROutput.java:close(526)) - 5CB09D6B115F4F799DB43E3C7AFDDA39 closed
2016-05-18 01:47:53,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:53 Completed running task attempt: attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,196 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:53,196 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536072863_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:53,197 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536072863_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536073188, Process_Duration=7, Process_End_Time=1463536073195, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:53,198 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536072863_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,198 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,198 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:53,198 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, taskAttemptId=attempt_1463536072863_0001_1_01_000000_0, creationTime=1463536073146, allocationTime=1463536073148, startTime=1463536073150, finishTime=1463536073199, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:53,199 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536072863_0001_1_01_000000_0
2016-05-18 01:47:53,200 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:TASK_FINISHED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, taskId=task_1463536072863_0001_1_01_000000, startTime=1463536073150, finishTime=1463536073199, timeTaken=49, status=SUCCEEDED, successfulAttemptID=attempt_1463536072863_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536073188, Process_Duration=7, Process_End_Time=1463536073195, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:53,200 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536072863_0001_00_000002
2016-05-18 01:47:53,200 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:53,200 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58]
2016-05-18 01:47:53,202 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:VERTEX_FINISHED]: vertexName=B5AF6EEFD1094908A33E4023ED163B58, vertexId=vertex_1463536072863_0001_1_01, initRequestedTime=1463536073054, initedTime=1463536073062, startRequestedTime=1463536073074, startedTime=1463536073074, finishTime=1463536073201, timeTaken=127, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536073188, Process_Duration=7, Process_End_Time=1463536073195, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536073150, firstTasksToStart=[ task_1463536072863_0001_1_01_000000 ], lastTaskFinishTime=1463536073199, lastTasksToFinish=[ task_1463536072863_0001_1_01_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536072863_0001_1_01_000000 ], longestDurationTasks=[ task_1463536072863_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:53,202 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:53,202 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536072863_0001_00_000002, stoppedTime=1463536073202, exitStatus=0
2016-05-18 01:47:53,202 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:53,203 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,203 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536072863_0001_1_00 [B2F07668EF9048C280EEE3C510CF4612]
2016-05-18 01:47:53,203 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536072863_0001_1
2016-05-18 01:47:53,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536072863_0001_1
2016-05-18 01:47:53,204 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 5CB09D6B115F4F799DB43E3C7AFDDA39 for vertex: vertex_1463536072863_0001_1_01 [B5AF6EEFD1094908A33E4023ED163B58], outputName: 5CB09D6B115F4F799DB43E3C7AFDDA39
2016-05-18 01:47:53,204 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072863_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:53,204 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536072863_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:53,211 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536072863_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:53,211 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536072863_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:53,211 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536072863_0001_00_000002, Exiting
2016-05-18 01:47:53,211 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536072863_0001_00_000002
2016-05-18 01:47:53,211 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536072863_0001_00_000002
2016-05-18 01:47:53,211 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536072863_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:53,211 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536072863_0001_00_000002 completed successfully
2016-05-18 01:47:53,214 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:5CB09D6B115F4F799DB43E3C7AFDDA39 of vertex/vertexGroup:B5AF6EEFD1094908A33E4023ED163B58 isVertexGroupOutput:false
2016-05-18 01:47:53,214 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,216 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536072863_0001_1, queueSize=0
2016-05-18 01:47:53,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536072863_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536072863_0001_1, startTime=1463536073050, finishTime=1463536073214, timeTaken=164, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072146314, Process_Duration=12, Process_End_Time=2927072146326, Read_Duration=1, Tuples_Read=74, Tuples_Written=74, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:53,217 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536072863_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:53,217 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536072863_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:53 Completed Dag: dag_1463536072863_0001_1
2016-05-18 01:47:53,218 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536072863_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:53,218 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536072863_0001_1_post
2016-05-18 01:47:53,219 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,219 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:53,219 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:53,219 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false, with id=dag_1463536072863_0001_1
2016-05-18 01:47:53,220 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4] comprehensivesort/(1/1) ...sive/true_null_true_false, with id=dag_1463536072863_0001_1
2016-05-18 01:47:53,225 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/true_null_true_false, applicationId=application_1463536072863_0001
2016-05-18 01:47:53,227 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:53,228 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:53,228 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:53,228 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:53,228 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:53,228 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:53,228 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:53,230 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:53,230 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:53,230 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:53,230 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:53,231 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:53,231 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:53,232 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:53,236 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/5747E269D3E44F979A163B28696C081E/E294B10E209643BCBAF7CFF55049DAB4/.tez/application_1463536072863_0001
2016-05-18 01:47:53,236 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false/_temporary
2016-05-18 01:47:53,237 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.396
2016-05-18 01:47:53,319 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:53,319 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:53,380 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 01:47:53,382 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:53,382 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:53,392 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.030
2016-05-18 01:47:53,393 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:53,393 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:53,394 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:53,418 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:53,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false
2016-05-18 01:47:53,482 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:53,483 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:53,484 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false"]
2016-05-18 01:47:53,484 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:53,484 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:53,485 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:53,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...e/false_false_false_false
2016-05-18 01:47:53,486 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:53,502 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:53,506 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:53,506 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:53,506 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:53,515 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001 doesn't exist and is created
2016-05-18 01:47:53,535 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:53,536 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:53,536 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd
2016-05-18 01:47:53,552 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536073506_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:53,552 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:53,552 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:53,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:53,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:53,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:53,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:53,557 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:53,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:53,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:53,559 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:53,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:53,560 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:53,560 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:53,561 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:53,562 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:53,562 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:53,562 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:53,563 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:53,563 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:53,566 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:53,567 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:53,567 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:53,567 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-log-dir
2016-05-18 01:47:53,568 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-log-dir/history.txt.appattempt_1463536073506_0001_000000, maxErrors=10
2016-05-18 01:47:53,569 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:53,569 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536073506_0001_000000, appSubmitTime=1463536073551, launchTime=1463536073551
2016-05-18 01:47:53,585 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:53,588 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56335
2016-05-18 01:47:53,590 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536073506
2016-05-18 01:47:53,592 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536073506_0001_000000, startTime=1463536073592
2016-05-18 01:47:53,592 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:53,636 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:53,636 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:53,636 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...e/false_false_false_false, applicationId=application_1463536073506_0001, dagName=[08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false
2016-05-18 01:47:53,656 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false
2016-05-18 01:47:53,656 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:53,657 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536073506_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-log-dir/dag_1463536073506_0001_1.dot
2016-05-18 01:47:53,658 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-log-dir/dag_1463536073506_0001_1-tez-dag.pb.txt
2016-05-18 01:47:53,671 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536073506_0001_1
2016-05-18 01:47:53,671 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,671 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false
2016-05-18 01:47:53 Running Dag: dag_1463536073506_0001_1
2016-05-18 01:47:53,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536073506_0001_1, submitTime=1463536073656
2016-05-18 01:47:53,689 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001/tez-conf.pb" } size: 40937 timestamp: 1463536073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:53,689 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,689 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:53,696 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001/tez-conf.pb" } size: 40937 timestamp: 1463536073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:53,696 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,696 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:53,697 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:53,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536073506_0001_1, initTime=1463536073697
2016-05-18 01:47:53,698 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536073506_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:53,698 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:53,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:DAG_STARTED]: dagID=dag_1463536073506_0001_1, startTime=1463536073699
2016-05-18 01:47:53,699 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536073506_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:53,699 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F00083BC3D4D4699A78DD1CBBFF238CD : {13F99B14020D4DB7908C04F8DC796032={InputName=13F99B14020D4DB7908C04F8DC796032}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:53,699 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 13F99B14020D4DB7908C04F8DC796032, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:53,699 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,699 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,699 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,700 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,700 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:53,700 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,701 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...e/false_false_false_false, applicationId=application_1463536073506_0001, dagName=[08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false
2016-05-18 01:47:53,703 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:53,703 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,703 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536073506_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:53,705 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415], numAdditionalOutputs=1
2016-05-18 01:47:53,705 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=1EDAF17287204DA6AB88822CC6DC0A76, vertex=vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:53,704 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 13F99B14020D4DB7908C04F8DC796032 on vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,707 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 13F99B14020D4DB7908C04F8DC796032 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:53,709 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:53,709 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:53,710 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:53,710 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:53,710 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E77DC7B4572447E8B253F1B412FBB415:1EDAF17287204DA6AB88822CC6DC0A76 using oldmapred API
2016-05-18 01:47:53,711 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=1EDAF17287204DA6AB88822CC6DC0A76, vertexName=E77DC7B4572447E8B253F1B412FBB415, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:53,711 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 13F99B14020D4DB7908C04F8DC796032 on vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, vertexId=vertex_1463536073506_0001_1_01, initRequestedTime=1463536073700, initedTime=1463536073705, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:53,712 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:53,713 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:53,713 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {13F99B14020D4DB7908C04F8DC796032=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:53,713 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] parallelism set to 1
2016-05-18 01:47:53,714 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:53,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, vertexId=vertex_1463536073506_0001_1_00, initRequestedTime=1463536073699, initedTime=1463536073714, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:53,715 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] with distanceFromRoot: 0
2016-05-18 01:47:53,715 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:53,715 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536073506_0001_1_00, startRequestedTime=1463536073700, startedTime=1463536073715
2016-05-18 01:47:53,716 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536073506_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:53,716 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:53,716 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536073506_0001_1_00 for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] numStartedSources: 1 numSources: 1
2016-05-18 01:47:53,716 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] with distanceFromRoot: 1
2016-05-18 01:47:53,717 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F00083BC3D4D4699A78DD1CBBFF238CD
2016-05-18 01:47:53,717 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536073506_0001_1_01, startRequestedTime=1463536073716, startedTime=1463536073717
2016-05-18 01:47:53,719 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:53,721 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E77DC7B4572447E8B253F1B412FBB415 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:53,722 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F00083BC3D4D4699A78DD1CBBFF238CD in vertex: E77DC7B4572447E8B253F1B412FBB415 numBipartiteSourceTasks: 1
2016-05-18 01:47:53,722 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_STARTED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, taskId=task_1463536073506_0001_1_00_000000, scheduledTime=1463536073720, launchTime=1463536073720
2016-05-18 01:47:53,722 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,723 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:53,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536073506_0001_00_000001, launchTime=1463536073725
2016-05-18 01:47:53,726 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536073506_0001_00_000001
2016-05-18 01:47:53,728 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536073506_0001_1_00_000000_0] started. Is using containerId: [container_1463536073506_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:53,728 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0, startTime=1463536073728, containerId=container_1463536073506_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:53,729 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536073506_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53 Starting to run new task attempt: attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,732 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,732 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,732 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:53,733 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false, VertexName: F00083BC3D4D4699A78DD1CBBFF238CD, VertexParallelism: 1, TaskAttemptID:attempt_1463536073506_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=13F99B14020D4DB7908C04F8DC796032, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E77DC7B4572447E8B253F1B412FBB415, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:53,733 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:53,734 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,738 INFO input.MRInput (MRInput.java:initialize(440)) - 13F99B14020D4DB7908C04F8DC796032 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:53,739 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 13F99B14020D4DB7908C04F8DC796032
2016-05-18 01:47:53,739 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536073506_0001, with dag id: dag_1463536073506_0001_1
2016-05-18 01:47:53,738 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536073506_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:53,739 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:53,740 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536073506_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:53,751 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F00083BC3D4D4699A78DD1CBBFF238CD, ordinal: 0
2016-05-18 01:47:53,751 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F00083BC3D4D4699A78DD1CBBFF238CD, mem on start (mb), free: 298, total: 356, max: 672
2016-05-18 01:47:53,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:53,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:53,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:53,752 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:53,752 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[13F99B14020D4DB7908C04F8DC796032:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E77DC7B4572447E8B253F1B412FBB415:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:53,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:53,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:53,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:53,753 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[13F99B14020D4DB7908C04F8DC796032:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E77DC7B4572447E8B253F1B412FBB415:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:53,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:53,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 13F99B14020D4DB7908C04F8DC796032
2016-05-18 01:47:53,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 13F99B14020D4DB7908C04F8DC796032 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:53,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:53,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:53,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:53,755 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,756 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:53,756 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 13F99B14020D4DB7908C04F8DC796032 initialized RecordReader from event
2016-05-18 01:47:53,760 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:53,769 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 13F99B14020D4DB7908C04F8DC796032
2016-05-18 01:47:53,769 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 7142A084FDBD41A9B86EC67D0C783FE1
2016-05-18 01:47:53,770 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7142A084FDBD41A9B86EC67D0C783FE1
2016-05-18 01:47:53,773 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E77DC7B4572447E8B253F1B412FBB415 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@5415cf91, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@63ee63b7, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:53,774 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E77DC7B4572447E8B253F1B412FBB415: , UsingHashComparator=false
2016-05-18 01:47:53,787 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:53,787 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:53,787 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E77DC7B4572447E8B253F1B412FBB415: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:53,788 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E77DC7B4572447E8B253F1B412FBB415: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:53,788 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:53,788 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F00083BC3D4D4699A78DD1CBBFF238CD, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:53,793 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,793 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F00083BC3D4D4699A78DD1CBBFF238CD, mem on close (mb), free: 197, total: 356, max: 672
2016-05-18 01:47:53,794 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E77DC7B4572447E8B253F1B412FBB415: Starting flush of map output
2016-05-18 01:47:53,794 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E77DC7B4572447E8B253F1B412FBB415: Span0.length = 37, perItem = 24
2016-05-18 01:47:53,795 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E77DC7B4572447E8B253F1B412FBB415: done sorting span=0, length=37, time=1
2016-05-18 01:47:53,795 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E77DC7B4572447E8B253F1B412FBB415: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:53,797 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E77DC7B4572447E8B253F1B412FBB415: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-local-dir/output/attempt_1463536073506_0001_1_00_000000_0_10047_0/file.out
2016-05-18 01:47:53,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:53 Completed running task attempt: attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:53,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536073506_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:53,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536073506_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536073788, Process_Duration=5, Process_End_Time=1463536073793, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:53,804 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536073506_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,804 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,805 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:53,805 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,805 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536073506_0001_00_000001
2016-05-18 01:47:53,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, taskAttemptId=attempt_1463536073506_0001_1_00_000000_0, creationTime=1463536073722, allocationTime=1463536073725, startTime=1463536073728, finishTime=1463536073804, timeTaken=76, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:53,806 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536073506_0001_1_00_000000_0
2016-05-18 01:47:53,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_FINISHED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, taskId=task_1463536073506_0001_1_00_000000, startTime=1463536073728, finishTime=1463536073806, timeTaken=78, status=SUCCEEDED, successfulAttemptID=attempt_1463536073506_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536073788, Process_Duration=5, Process_End_Time=1463536073793, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:53,807 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:53,807 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_FINISHED]: vertexName=F00083BC3D4D4699A78DD1CBBFF238CD, vertexId=vertex_1463536073506_0001_1_00, initRequestedTime=1463536073699, initedTime=1463536073714, startRequestedTime=1463536073700, startedTime=1463536073715, finishTime=1463536073807, timeTaken=92, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536073788, Process_Duration=5, Process_End_Time=1463536073793, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536073728, firstTasksToStart=[ task_1463536073506_0001_1_00_000000 ], lastTaskFinishTime=1463536073804, lastTasksToFinish=[ task_1463536073506_0001_1_00_000000 ], minTaskDuration=76, maxTaskDuration=76, avgTaskDuration=76.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536073506_0001_1_00_000000 ], longestDurationTasks=[ task_1463536073506_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:53,808 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:53,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536073506_0001_00_000001, stoppedTime=1463536073809, exitStatus=0
2016-05-18 01:47:53,809 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] attempt: attempt_1463536073506_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:53,809 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:53,809 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E77DC7B4572447E8B253F1B412FBB415
2016-05-18 01:47:53,809 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,809 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F00083BC3D4D4699A78DD1CBBFF238CD, destinationVertexName=E77DC7B4572447E8B253F1B412FBB415. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:53,810 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536073506_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:53,810 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,810 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_STARTED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, taskId=task_1463536073506_0001_1_01_000000, scheduledTime=1463536073810, launchTime=1463536073810
2016-05-18 01:47:53,812 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536073506_0001_00_000002, launchTime=1463536073814
2016-05-18 01:47:53,815 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536073506_0001_00_000002
2016-05-18 01:47:53,815 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536073506_0001_1_01_000000_0] started. Is using containerId: [container_1463536073506_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:53,815 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0, startTime=1463536073815, containerId=container_1463536073506_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:53,816 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536073506_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53 Starting to run new task attempt: attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,816 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,817 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,817 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:53,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false, VertexName: E77DC7B4572447E8B253F1B412FBB415, VertexParallelism: 1, TaskAttemptID:attempt_1463536073506_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F00083BC3D4D4699A78DD1CBBFF238CD, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=1EDAF17287204DA6AB88822CC6DC0A76, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:53,818 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:53,817 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536073506_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:53,818 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536073506_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:53,818 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,818 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536073506_0001_00_000001, Exiting
2016-05-18 01:47:53,818 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536073506_0001_00_000001
2016-05-18 01:47:53,819 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536073506_0001_00_000001
2016-05-18 01:47:53,819 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536073506_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:53,820 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536073506_0001_00_000001 completed successfully
2016-05-18 01:47:53,824 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:53,825 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:53,826 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536073506_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,826 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536073506_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:53,832 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E77DC7B4572447E8B253F1B412FBB415, ordinal: 1
2016-05-18 01:47:53,832 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E77DC7B4572447E8B253F1B412FBB415, mem on start (mb), free: 190, total: 356, max: 672
2016-05-18 01:47:53,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:53,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:53,833 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:53,834 INFO output.MROutput (MROutput.java:initialize(421)) - 1EDAF17287204DA6AB88822CC6DC0A76: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:53,835 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:53,835 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[1EDAF17287204DA6AB88822CC6DC0A76:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [F00083BC3D4D4699A78DD1CBBFF238CD:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:53,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:53,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:53,835 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:53,836 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[1EDAF17287204DA6AB88822CC6DC0A76:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [F00083BC3D4D4699A78DD1CBBFF238CD:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:53,836 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:53,836 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F00083BC3D4D4699A78DD1CBBFF238CD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:53,837 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F00083BC3D4D4699A78DD1CBBFF238CD: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:53,837 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F00083BC3D4D4699A78DD1CBBFF238CD: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:53,838 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F00083BC3D4D4699A78DD1CBBFF238CD with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:53,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:53,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:53,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F00083BC3D4D4699A78DD1CBBFF238CD
2016-05-18 01:47:53,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:53,840 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,841 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F00083BC3D4D4699A78DD1CBBFF238CD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:53,842 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536073506_0001_1_00_000000_0_10047} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536073842, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:47:53,843 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F00083BC3D4D4699A78DD1CBBFF238CD
2016-05-18 01:47:53,843 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:53,843 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F00083BC3D4D4699A78DD1CBBFF238CD, wasInterrupted=false
2016-05-18 01:47:53,844 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:53,844 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:53,845 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001_wd/localmode-local-dir/output/attempt_1463536073506_0001_1_00_000000_0_10047/file.out, len=732, isLocal=true
2016-05-18 01:47:53,845 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:53,845 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:53,845 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:53,846 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:53,846 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F00083BC3D4D4699A78DD1CBBFF238CD
2016-05-18 01:47:53,846 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F00083BC3D4D4699A78DD1CBBFF238CD: Shuffle Runner thread complete
2016-05-18 01:47:53,846 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:53,847 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 7142A084FDBD41A9B86EC67D0C783FE1
2016-05-18 01:47:53,847 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false"], id: 1EDAF17287204DA6AB88822CC6DC0A76
2016-05-18 01:47:53,848 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false"]
2016-05-18 01:47:53,848 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7142A084FDBD41A9B86EC67D0C783FE1, for 1 inputs
2016-05-18 01:47:53,848 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E77DC7B4572447E8B253F1B412FBB415, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:53,850 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F00083BC3D4D4699A78DD1CBBFF238CD: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:53,857 INFO output.MROutput (MROutput.java:close(526)) - 1EDAF17287204DA6AB88822CC6DC0A76 closed
2016-05-18 01:47:53,857 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536073506_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:53,858 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,859 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E77DC7B4572447E8B253F1B412FBB415, mem on close (mb), free: 187, total: 356, max: 672
2016-05-18 01:47:53,859 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F00083BC3D4D4699A78DD1CBBFF238CD
2016-05-18 01:47:53,859 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F00083BC3D4D4699A78DD1CBBFF238CD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:53,860 INFO output.MROutput (MROutput.java:close(526)) - 1EDAF17287204DA6AB88822CC6DC0A76 closed
2016-05-18 01:47:53,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:53 Completed running task attempt: attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,861 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:53,861 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536073506_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:53,862 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536073506_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536073848, Process_Duration=10, Process_End_Time=1463536073858, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:53,863 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536073506_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, taskAttemptId=attempt_1463536073506_0001_1_01_000000_0, creationTime=1463536073812, allocationTime=1463536073813, startTime=1463536073815, finishTime=1463536073863, timeTaken=48, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:53,863 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:53,864 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:53,864 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,864 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536073506_0001_00_000002
2016-05-18 01:47:53,864 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536073506_0001_1_01_000000_0
2016-05-18 01:47:53,865 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:TASK_FINISHED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, taskId=task_1463536073506_0001_1_01_000000, startTime=1463536073815, finishTime=1463536073865, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463536073506_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536073848, Process_Duration=10, Process_End_Time=1463536073858, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:53,865 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:53,866 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415]
2016-05-18 01:47:53,867 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:VERTEX_FINISHED]: vertexName=E77DC7B4572447E8B253F1B412FBB415, vertexId=vertex_1463536073506_0001_1_01, initRequestedTime=1463536073700, initedTime=1463536073705, startRequestedTime=1463536073716, startedTime=1463536073717, finishTime=1463536073866, timeTaken=149, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536073848, Process_Duration=10, Process_End_Time=1463536073858, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536073815, firstTasksToStart=[ task_1463536073506_0001_1_01_000000 ], lastTaskFinishTime=1463536073863, lastTasksToFinish=[ task_1463536073506_0001_1_01_000000 ], minTaskDuration=48, maxTaskDuration=48, avgTaskDuration=48.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536073506_0001_1_01_000000 ], longestDurationTasks=[ task_1463536073506_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:53,867 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:53,867 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536073506_0001_00_000002, stoppedTime=1463536073867, exitStatus=0
2016-05-18 01:47:53,867 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:53,867 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,867 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536073506_0001_1_00 [F00083BC3D4D4699A78DD1CBBFF238CD]
2016-05-18 01:47:53,868 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536073506_0001_1
2016-05-18 01:47:53,868 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536073506_0001_1
2016-05-18 01:47:53,868 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 1EDAF17287204DA6AB88822CC6DC0A76 for vertex: vertex_1463536073506_0001_1_01 [E77DC7B4572447E8B253F1B412FBB415], outputName: 1EDAF17287204DA6AB88822CC6DC0A76
2016-05-18 01:47:53,871 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536073506_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:53,871 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536073506_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:53,875 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536073506_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:53,875 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536073506_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:53,875 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536073506_0001_00_000002, Exiting
2016-05-18 01:47:53,875 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536073506_0001_00_000002
2016-05-18 01:47:53,876 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536073506_0001_00_000002
2016-05-18 01:47:53,876 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536073506_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:53,876 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536073506_0001_00_000002 completed successfully
2016-05-18 01:47:53,879 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:1EDAF17287204DA6AB88822CC6DC0A76 of vertex/vertexGroup:E77DC7B4572447E8B253F1B412FBB415 isVertexGroupOutput:false
2016-05-18 01:47:53,879 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:53,881 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536073506_0001_1, queueSize=0
2016-05-18 01:47:53,881 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536073506_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536073506_0001_1, startTime=1463536073698, finishTime=1463536073879, timeTaken=181, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072147636, Process_Duration=15, Process_End_Time=2927072147651, Read_Duration=3, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:53,881 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536073506_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:53,881 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536073506_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:53 Completed Dag: dag_1463536073506_0001_1
2016-05-18 01:47:53,882 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536073506_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:53,882 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536073506_0001_1_post
2016-05-18 01:47:53,882 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:53,882 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:53,882 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:53,882 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false, with id=dag_1463536073506_0001_1
2016-05-18 01:47:53,882 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491] comprehensivesort/(1/1) ...e/false_false_false_false, with id=dag_1463536073506_0001_1
2016-05-18 01:47:53,892 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...e/false_false_false_false, applicationId=application_1463536073506_0001
2016-05-18 01:47:53,894 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:53,894 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:53,894 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:53,894 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:53,894 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:53,895 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:53,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:53,896 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:53,896 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:53,897 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:53,898 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:53,898 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:53,898 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:53,899 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:53,904 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/08366B4A3D19430EA8E27C41B90F4F76/BB135DB9570C432DBEC2BEA9BF12A491/.tez/application_1463536073506_0001
2016-05-18 01:47:53,905 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false/_temporary
2016-05-18 01:47:53,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.423
2016-05-18 01:47:53,981 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:53,981 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:54,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 01:47:54,034 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.022
2016-05-18 01:47:54,035 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:54,035 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:54,035 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:54,054 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:54,062 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false"]
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:54,118 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:54,120 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ve/true_false_false_false
2016-05-18 01:47:54,120 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:54,134 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:54,139 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:54,140 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:54,140 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:54,149 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001 doesn't exist and is created
2016-05-18 01:47:54,170 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:54,170 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:54,171 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd
2016-05-18 01:47:54,185 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536074140_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:54,186 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:54,186 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:54,188 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:54,188 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:54,188 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:54,188 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:54,189 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:54,191 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:54,192 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:54,192 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:54,192 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:54,192 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:54,192 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:54,192 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:54,193 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:54,193 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:54,193 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:54,193 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:54,193 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:54,195 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:54,196 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:54,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:54,196 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-log-dir
2016-05-18 01:47:54,197 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-log-dir/history.txt.appattempt_1463536074140_0001_000000, maxErrors=10
2016-05-18 01:47:54,197 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:54,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536074140_0001_000000, appSubmitTime=1463536074185, launchTime=1463536074185
2016-05-18 01:47:54,211 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:54,217 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:43463
2016-05-18 01:47:54,218 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536074140
2016-05-18 01:47:54,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536074140_0001_000000, startTime=1463536074220
2016-05-18 01:47:54,220 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:54,270 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:54,271 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:54,271 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ve/true_false_false_false, applicationId=application_1463536074140_0001, dagName=[2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false
2016-05-18 01:47:54,291 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false
2016-05-18 01:47:54,291 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:54,292 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536074140_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-log-dir/dag_1463536074140_0001_1.dot
2016-05-18 01:47:54,292 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-log-dir/dag_1463536074140_0001_1-tez-dag.pb.txt
2016-05-18 01:47:54,299 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536074140_0001_1
2016-05-18 01:47:54,299 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,299 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false
2016-05-18 01:47:54 Running Dag: dag_1463536074140_0001_1
2016-05-18 01:47:54,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536074140_0001_1, submitTime=1463536074291
2016-05-18 01:47:54,317 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001/tez-conf.pb" } size: 40937 timestamp: 1463536074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:54,317 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,317 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:54,318 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001/tez-conf.pb" } size: 40937 timestamp: 1463536074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:54,318 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,319 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:54,319 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:54,320 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536074140_0001_1, initTime=1463536074319
2016-05-18 01:47:54,320 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074140_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:54,320 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:54,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:DAG_STARTED]: dagID=dag_1463536074140_0001_1, startTime=1463536074321
2016-05-18 01:47:54,321 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074140_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:54,321 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D2AB062955564CE381CFF735E822D65C : {0C0000BED6E04CC5A207988D04639FCD={InputName=0C0000BED6E04CC5A207988D04639FCD}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:54,321 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 0C0000BED6E04CC5A207988D04639FCD, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:54,321 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,321 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,322 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,322 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,322 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:54,323 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 0C0000BED6E04CC5A207988D04639FCD on vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,323 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,324 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 0C0000BED6E04CC5A207988D04639FCD asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:54,326 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:54,326 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,327 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ve/true_false_false_false, applicationId=application_1463536074140_0001, dagName=[2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false
2016-05-18 01:47:54,327 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:54,328 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:54,328 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:54,328 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:54,329 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 0C0000BED6E04CC5A207988D04639FCD on vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,329 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536074140_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:54,331 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0], numAdditionalOutputs=1
2016-05-18 01:47:54,331 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=16C560E689744CFABADEE99F3C6B9513, vertex=vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:54,334 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F2D698203B074F559183398566AB3CE0:16C560E689744CFABADEE99F3C6B9513 using oldmapred API
2016-05-18 01:47:54,334 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=16C560E689744CFABADEE99F3C6B9513, vertexName=F2D698203B074F559183398566AB3CE0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:54,335 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F2D698203B074F559183398566AB3CE0, vertexId=vertex_1463536074140_0001_1_01, initRequestedTime=1463536074323, initedTime=1463536074331, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:54,336 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:54,336 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:54,337 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {0C0000BED6E04CC5A207988D04639FCD=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:54,337 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] parallelism set to 1
2016-05-18 01:47:54,337 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:54,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D2AB062955564CE381CFF735E822D65C, vertexId=vertex_1463536074140_0001_1_00, initRequestedTime=1463536074321, initedTime=1463536074338, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:54,339 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] with distanceFromRoot: 0
2016-05-18 01:47:54,339 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:54,339 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536074140_0001_1_00, startRequestedTime=1463536074323, startedTime=1463536074339
2016-05-18 01:47:54,340 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536074140_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:54,340 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:54,341 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536074140_0001_1_00 for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] numStartedSources: 1 numSources: 1
2016-05-18 01:47:54,341 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] with distanceFromRoot: 1
2016-05-18 01:47:54,341 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D2AB062955564CE381CFF735E822D65C
2016-05-18 01:47:54,342 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,342 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536074140_0001_1_01, startRequestedTime=1463536074341, startedTime=1463536074341
2016-05-18 01:47:54,343 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:54,345 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F2D698203B074F559183398566AB3CE0 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:54,345 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_STARTED]: vertexName=D2AB062955564CE381CFF735E822D65C, taskId=task_1463536074140_0001_1_00_000000, scheduledTime=1463536074345, launchTime=1463536074345
2016-05-18 01:47:54,345 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D2AB062955564CE381CFF735E822D65C in vertex: F2D698203B074F559183398566AB3CE0 numBipartiteSourceTasks: 1
2016-05-18 01:47:54,346 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536074140_0001, with dag id: dag_1463536074140_0001_1
2016-05-18 01:47:54,346 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,347 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:54,350 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536074140_0001_00_000001, launchTime=1463536074349
2016-05-18 01:47:54,350 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074140_0001_00_000001
2016-05-18 01:47:54,351 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536074140_0001_1_00_000000_0] started. Is using containerId: [container_1463536074140_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:54,351 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D2AB062955564CE381CFF735E822D65C, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0, startTime=1463536074351, containerId=container_1463536074140_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:54,352 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074140_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54 Starting to run new task attempt: attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,353 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,353 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,354 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:54,354 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false, VertexName: D2AB062955564CE381CFF735E822D65C, VertexParallelism: 1, TaskAttemptID:attempt_1463536074140_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=0C0000BED6E04CC5A207988D04639FCD, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F2D698203B074F559183398566AB3CE0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:54,354 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:54,355 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,356 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536074140_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:54,356 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536074140_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:54,359 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:54,361 INFO input.MRInput (MRInput.java:initialize(440)) - 0C0000BED6E04CC5A207988D04639FCD using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:54,361 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 0C0000BED6E04CC5A207988D04639FCD
2016-05-18 01:47:54,369 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D2AB062955564CE381CFF735E822D65C, ordinal: 0
2016-05-18 01:47:54,369 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2AB062955564CE381CFF735E822D65C, mem on start (mb), free: 296, total: 355, max: 672
2016-05-18 01:47:54,369 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:54,369 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:54,369 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:54,369 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:54,369 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[0C0000BED6E04CC5A207988D04639FCD:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F2D698203B074F559183398566AB3CE0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:54,370 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:54,370 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:54,370 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:54,370 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[0C0000BED6E04CC5A207988D04639FCD:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F2D698203B074F559183398566AB3CE0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:54,370 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:54,371 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 0C0000BED6E04CC5A207988D04639FCD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:54,371 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:54,371 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:54,371 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 0C0000BED6E04CC5A207988D04639FCD
2016-05-18 01:47:54,372 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:54,372 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,375 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:54,375 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:54,376 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 0C0000BED6E04CC5A207988D04639FCD initialized RecordReader from event
2016-05-18 01:47:54,376 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 0C0000BED6E04CC5A207988D04639FCD
2016-05-18 01:47:54,376 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: C9A3693823CF4C1B82E055B47BCE9F1A
2016-05-18 01:47:54,377 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] C9A3693823CF4C1B82E055B47BCE9F1A
2016-05-18 01:47:54,381 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F2D698203B074F559183398566AB3CE0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@464eb55e, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@b038c21, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:54,382 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F2D698203B074F559183398566AB3CE0: , UsingHashComparator=false
2016-05-18 01:47:54,395 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:54,395 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:54,396 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F2D698203B074F559183398566AB3CE0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:54,396 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F2D698203B074F559183398566AB3CE0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:54,396 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:54,397 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D2AB062955564CE381CFF735E822D65C, all 1 inputs ready in: 00:00:00.001
2016-05-18 01:47:54,401 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,402 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D2AB062955564CE381CFF735E822D65C, mem on close (mb), free: 193, total: 355, max: 672
2016-05-18 01:47:54,402 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F2D698203B074F559183398566AB3CE0: Starting flush of map output
2016-05-18 01:47:54,402 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F2D698203B074F559183398566AB3CE0: Span0.length = 37, perItem = 24
2016-05-18 01:47:54,404 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F2D698203B074F559183398566AB3CE0: done sorting span=0, length=37, time=2
2016-05-18 01:47:54,404 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F2D698203B074F559183398566AB3CE0: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:54,405 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F2D698203B074F559183398566AB3CE0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-local-dir/output/attempt_1463536074140_0001_1_00_000000_0_10053_0/file.out
2016-05-18 01:47:54,410 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:54 Completed running task attempt: attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,411 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:54,411 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536074140_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:54,412 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536074140_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536074397, Process_Duration=4, Process_End_Time=1463536074401, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:54,413 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536074140_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:54,413 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:54,413 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D2AB062955564CE381CFF735E822D65C, taskAttemptId=attempt_1463536074140_0001_1_00_000000_0, creationTime=1463536074346, allocationTime=1463536074349, startTime=1463536074351, finishTime=1463536074413, timeTaken=62, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:54,413 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:54,413 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,413 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536074140_0001_1_00_000000_0
2016-05-18 01:47:54,414 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_FINISHED]: vertexName=D2AB062955564CE381CFF735E822D65C, taskId=task_1463536074140_0001_1_00_000000, startTime=1463536074351, finishTime=1463536074414, timeTaken=63, status=SUCCEEDED, successfulAttemptID=attempt_1463536074140_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536074397, Process_Duration=4, Process_End_Time=1463536074401, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:54,415 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:54,415 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,416 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_FINISHED]: vertexName=D2AB062955564CE381CFF735E822D65C, vertexId=vertex_1463536074140_0001_1_00, initRequestedTime=1463536074321, initedTime=1463536074338, startRequestedTime=1463536074323, startedTime=1463536074339, finishTime=1463536074415, timeTaken=76, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536074397, Process_Duration=4, Process_End_Time=1463536074401, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536074351, firstTasksToStart=[ task_1463536074140_0001_1_00_000000 ], lastTaskFinishTime=1463536074413, lastTasksToFinish=[ task_1463536074140_0001_1_00_000000 ], minTaskDuration=62, maxTaskDuration=62, avgTaskDuration=62.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536074140_0001_1_00_000000 ], longestDurationTasks=[ task_1463536074140_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:54,416 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:54,417 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536074140_0001_00_000001, stoppedTime=1463536074417, exitStatus=0
2016-05-18 01:47:54,417 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] attempt: attempt_1463536074140_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:54,417 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F2D698203B074F559183398566AB3CE0
2016-05-18 01:47:54,417 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074140_0001_00_000001
2016-05-18 01:47:54,418 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:54,417 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D2AB062955564CE381CFF735E822D65C, destinationVertexName=F2D698203B074F559183398566AB3CE0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:54,419 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,420 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:54,420 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536074140_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:54,421 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_STARTED]: vertexName=F2D698203B074F559183398566AB3CE0, taskId=task_1463536074140_0001_1_01_000000, scheduledTime=1463536074421, launchTime=1463536074421
2016-05-18 01:47:54,421 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,418 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536074140_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:54,422 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074140_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:54,422 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536074140_0001_00_000001, Exiting
2016-05-18 01:47:54,423 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536074140_0001_00_000001
2016-05-18 01:47:54,423 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536074140_0001_00_000001
2016-05-18 01:47:54,426 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536074140_0001_00_000002, launchTime=1463536074425
2016-05-18 01:47:54,426 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536074140_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:54,426 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536074140_0001_00_000001 completed successfully
2016-05-18 01:47:54,427 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074140_0001_00_000002
2016-05-18 01:47:54,427 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536074140_0001_1_01_000000_0] started. Is using containerId: [container_1463536074140_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:54,427 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F2D698203B074F559183398566AB3CE0, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0, startTime=1463536074427, containerId=container_1463536074140_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:54,428 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074140_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54 Starting to run new task attempt: attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,428 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,429 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,429 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:54,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false, VertexName: F2D698203B074F559183398566AB3CE0, VertexParallelism: 1, TaskAttemptID:attempt_1463536074140_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D2AB062955564CE381CFF735E822D65C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=16C560E689744CFABADEE99F3C6B9513, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:54,429 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:54,429 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,430 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536074140_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,430 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536074140_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:54,433 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:54,433 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:54,441 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F2D698203B074F559183398566AB3CE0, ordinal: 1
2016-05-18 01:47:54,441 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2D698203B074F559183398566AB3CE0, mem on start (mb), free: 187, total: 355, max: 672
2016-05-18 01:47:54,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:54,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:54,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:54,445 INFO output.MROutput (MROutput.java:initialize(421)) - 16C560E689744CFABADEE99F3C6B9513: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:54,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:54,446 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D2AB062955564CE381CFF735E822D65C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [16C560E689744CFABADEE99F3C6B9513:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:54,446 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:54,446 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:54,446 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:54,446 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D2AB062955564CE381CFF735E822D65C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [16C560E689744CFABADEE99F3C6B9513:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:54,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:54,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D2AB062955564CE381CFF735E822D65C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:54,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:54,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:54,447 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D2AB062955564CE381CFF735E822D65C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:54,447 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D2AB062955564CE381CFF735E822D65C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:54,448 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D2AB062955564CE381CFF735E822D65C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:54,449 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D2AB062955564CE381CFF735E822D65C
2016-05-18 01:47:54,450 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:54,450 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,451 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D2AB062955564CE381CFF735E822D65C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:54,453 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536074140_0001_1_00_000000_0_10053} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536074453, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:47:54,453 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D2AB062955564CE381CFF735E822D65C
2016-05-18 01:47:54,453 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:54,453 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D2AB062955564CE381CFF735E822D65C, wasInterrupted=false
2016-05-18 01:47:54,454 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - D2AB062955564CE381CFF735E822D65C: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:54,455 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:54,455 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: C9A3693823CF4C1B82E055B47BCE9F1A
2016-05-18 01:47:54,455 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false"], id: 16C560E689744CFABADEE99F3C6B9513
2016-05-18 01:47:54,455 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false"]
2016-05-18 01:47:54,456 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] C9A3693823CF4C1B82E055B47BCE9F1A, for 1 inputs
2016-05-18 01:47:54,456 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:54,456 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:54,457 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001_wd/localmode-local-dir/output/attempt_1463536074140_0001_1_00_000000_0_10053/file.out, len=732, isLocal=true
2016-05-18 01:47:54,457 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:54,457 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:54,457 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:54,458 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:54,458 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D2AB062955564CE381CFF735E822D65C
2016-05-18 01:47:54,458 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D2AB062955564CE381CFF735E822D65C: Shuffle Runner thread complete
2016-05-18 01:47:54,458 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F2D698203B074F559183398566AB3CE0, all 1 inputs ready in: 00:00:00.002
2016-05-18 01:47:54,459 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D2AB062955564CE381CFF735E822D65C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:54,466 INFO output.MROutput (MROutput.java:close(526)) - 16C560E689744CFABADEE99F3C6B9513 closed
2016-05-18 01:47:54,466 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536074140_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:54,467 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,467 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2D698203B074F559183398566AB3CE0, mem on close (mb), free: 183, total: 355, max: 672
2016-05-18 01:47:54,467 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D2AB062955564CE381CFF735E822D65C
2016-05-18 01:47:54,467 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D2AB062955564CE381CFF735E822D65C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:54,467 INFO output.MROutput (MROutput.java:close(526)) - 16C560E689744CFABADEE99F3C6B9513 closed
2016-05-18 01:47:54,467 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:54 Completed running task attempt: attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,468 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:54,468 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536074140_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:54,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536074140_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536074458, Process_Duration=9, Process_End_Time=1463536074467, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:54,469 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536074140_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:54,469 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:54,469 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:54,469 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,470 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074140_0001_00_000002
2016-05-18 01:47:54,470 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F2D698203B074F559183398566AB3CE0, taskAttemptId=attempt_1463536074140_0001_1_01_000000_0, creationTime=1463536074421, allocationTime=1463536074423, startTime=1463536074427, finishTime=1463536074470, timeTaken=43, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:54,470 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536074140_0001_1_01_000000_0
2016-05-18 01:47:54,471 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:TASK_FINISHED]: vertexName=F2D698203B074F559183398566AB3CE0, taskId=task_1463536074140_0001_1_01_000000, startTime=1463536074427, finishTime=1463536074470, timeTaken=43, status=SUCCEEDED, successfulAttemptID=attempt_1463536074140_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536074458, Process_Duration=9, Process_End_Time=1463536074467, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:54,471 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:54,471 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0]
2016-05-18 01:47:54,473 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:VERTEX_FINISHED]: vertexName=F2D698203B074F559183398566AB3CE0, vertexId=vertex_1463536074140_0001_1_01, initRequestedTime=1463536074323, initedTime=1463536074331, startRequestedTime=1463536074341, startedTime=1463536074341, finishTime=1463536074471, timeTaken=130, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536074458, Process_Duration=9, Process_End_Time=1463536074467, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536074427, firstTasksToStart=[ task_1463536074140_0001_1_01_000000 ], lastTaskFinishTime=1463536074470, lastTasksToFinish=[ task_1463536074140_0001_1_01_000000 ], minTaskDuration=43, maxTaskDuration=43, avgTaskDuration=43.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536074140_0001_1_01_000000 ], longestDurationTasks=[ task_1463536074140_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:54,473 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:54,473 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536074140_0001_00_000002, stoppedTime=1463536074473, exitStatus=0
2016-05-18 01:47:54,473 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:54,473 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:54,474 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536074140_0001_1_00 [D2AB062955564CE381CFF735E822D65C]
2016-05-18 01:47:54,474 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536074140_0001_1
2016-05-18 01:47:54,474 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536074140_0001_1
2016-05-18 01:47:54,474 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 16C560E689744CFABADEE99F3C6B9513 for vertex: vertex_1463536074140_0001_1_01 [F2D698203B074F559183398566AB3CE0], outputName: 16C560E689744CFABADEE99F3C6B9513
2016-05-18 01:47:54,474 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074140_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:54,475 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536074140_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:54,481 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536074140_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:54,481 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074140_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:54,482 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536074140_0001_00_000002, Exiting
2016-05-18 01:47:54,482 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536074140_0001_00_000002
2016-05-18 01:47:54,482 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536074140_0001_00_000002
2016-05-18 01:47:54,482 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536074140_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:54,482 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536074140_0001_00_000002 completed successfully
2016-05-18 01:47:54,484 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:16C560E689744CFABADEE99F3C6B9513 of vertex/vertexGroup:F2D698203B074F559183398566AB3CE0 isVertexGroupOutput:false
2016-05-18 01:47:54,484 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:54,486 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536074140_0001_1, queueSize=0
2016-05-18 01:47:54,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074140_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536074140_0001_1, startTime=1463536074320, finishTime=1463536074484, timeTaken=164, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=280, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072148855, Process_Duration=13, Process_End_Time=2927072148868, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:54,487 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536074140_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:54,487 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074140_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:54 Completed Dag: dag_1463536074140_0001_1
2016-05-18 01:47:54,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536074140_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:54,488 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536074140_0001_1_post
2016-05-18 01:47:54,488 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:54,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:54,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false, with id=dag_1463536074140_0001_1
2016-05-18 01:47:54,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36] comprehensivesort/(1/1) ...ve/true_false_false_false, with id=dag_1463536074140_0001_1
2016-05-18 01:47:54,494 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ve/true_false_false_false, applicationId=application_1463536074140_0001
2016-05-18 01:47:54,497 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:54,497 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:54,497 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:54,497 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:54,498 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:54,498 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:54,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:54,499 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:54,499 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:54,499 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:54,500 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:54,500 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:54,501 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:54,503 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:54,508 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2DC920F5C34B4D028AA0857F0F4D0681/EE1EB7E0BE9046FC9EC6E43ACEDF6E36/.tez/application_1463536074140_0001
2016-05-18 01:47:54,509 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false/_temporary
2016-05-18 01:47:54,509 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.390
2016-05-18 01:47:54,587 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:54,587 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:54,628 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 01:47:54,645 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.028
2016-05-18 01:47:54,645 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:54,646 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:54,646 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:54,670 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:54,679 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false
2016-05-18 01:47:54,740 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:54,741 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:54,742 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false"]
2016-05-18 01:47:54,742 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:54,743 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:54,743 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:54,744 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/true_true_false_false
2016-05-18 01:47:54,744 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:54,758 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:54,762 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:54,763 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:54,763 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:54,772 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001 doesn't exist and is created
2016-05-18 01:47:54,791 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:54,791 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:54,792 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd
2016-05-18 01:47:54,808 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536074763_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:54,808 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:54,808 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:54,810 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:54,810 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:54,810 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:54,811 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:54,812 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:54,819 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:54,819 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:54,819 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:54,820 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:54,820 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:54,820 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:54,821 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:54,821 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:54,821 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:54,821 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:54,822 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:54,822 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:54,824 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:54,825 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:54,825 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:54,825 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-log-dir
2016-05-18 01:47:54,826 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-log-dir/history.txt.appattempt_1463536074763_0001_000000, maxErrors=10
2016-05-18 01:47:54,826 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:54,826 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536074763_0001_000000, appSubmitTime=1463536074807, launchTime=1463536074807
2016-05-18 01:47:54,833 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:54,835 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:37206
2016-05-18 01:47:54,836 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536074763
2016-05-18 01:47:54,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536074763_0001_000000, startTime=1463536074838
2016-05-18 01:47:54,838 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:54,892 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:54,892 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:54,892 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/true_true_false_false, applicationId=application_1463536074763_0001, dagName=[2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false
2016-05-18 01:47:54,911 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false
2016-05-18 01:47:54,911 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:54,912 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536074763_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-log-dir/dag_1463536074763_0001_1.dot
2016-05-18 01:47:54,913 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-log-dir/dag_1463536074763_0001_1-tez-dag.pb.txt
2016-05-18 01:47:54,919 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536074763_0001_1
2016-05-18 01:47:54,920 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,920 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false
2016-05-18 01:47:54 Running Dag: dag_1463536074763_0001_1
2016-05-18 01:47:54,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536074763_0001_1, submitTime=1463536074911
2016-05-18 01:47:54,936 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001/tez-conf.pb" } size: 40937 timestamp: 1463536074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:54,937 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,937 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:54,938 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001/tez-conf.pb" } size: 40937 timestamp: 1463536074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:54,938 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:54,939 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:54,939 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:54,940 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536074763_0001_1, initTime=1463536074940
2016-05-18 01:47:54,941 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074763_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:54,941 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:54,941 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:DAG_STARTED]: dagID=dag_1463536074763_0001_1, startTime=1463536074941
2016-05-18 01:47:54,942 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/true_true_false_false, applicationId=application_1463536074763_0001, dagName=[2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false
2016-05-18 01:47:54,942 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074763_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:54,942 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: EC20C9688ADF47DAAA35F5BD8FF80D87 : {63C908ACCEF44FE0BC97C79F46C7D3D1={InputName=63C908ACCEF44FE0BC97C79F46C7D3D1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:54,942 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 63C908ACCEF44FE0BC97C79F46C7D3D1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:54,942 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,942 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,943 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,943 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,943 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:54,944 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:54,945 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 63C908ACCEF44FE0BC97C79F46C7D3D1 on vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,947 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:54,947 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:54,948 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:54,948 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 63C908ACCEF44FE0BC97C79F46C7D3D1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:54,948 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536074763_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:54,949 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557], numAdditionalOutputs=1
2016-05-18 01:47:54,949 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=FF605BAF830F479BB43B6C39C96A6300, vertex=vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:54,951 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:54,951 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:54,951 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:54,951 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:54,952 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 63C908ACCEF44FE0BC97C79F46C7D3D1 on vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,952 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F243FE12DA264862873D0AE0CF2FC557:FF605BAF830F479BB43B6C39C96A6300 using oldmapred API
2016-05-18 01:47:54,952 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=FF605BAF830F479BB43B6C39C96A6300, vertexName=F243FE12DA264862873D0AE0CF2FC557, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:54,954 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, vertexId=vertex_1463536074763_0001_1_01, initRequestedTime=1463536074944, initedTime=1463536074949, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:54,954 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:54,955 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:54,955 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {63C908ACCEF44FE0BC97C79F46C7D3D1=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:54,956 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] parallelism set to 1
2016-05-18 01:47:54,956 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:54,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, vertexId=vertex_1463536074763_0001_1_00, initRequestedTime=1463536074942, initedTime=1463536074956, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:54,957 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] with distanceFromRoot: 0
2016-05-18 01:47:54,957 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:54,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536074763_0001_1_00, startRequestedTime=1463536074944, startedTime=1463536074957
2016-05-18 01:47:54,958 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536074763_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:54,958 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:54,958 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536074763_0001_1_00 for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] numStartedSources: 1 numSources: 1
2016-05-18 01:47:54,958 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in EC20C9688ADF47DAAA35F5BD8FF80D87
2016-05-18 01:47:54,959 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,959 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] with distanceFromRoot: 1
2016-05-18 01:47:54,962 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536074763_0001_1_01, startRequestedTime=1463536074958, startedTime=1463536074959
2016-05-18 01:47:54,962 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:54,963 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F243FE12DA264862873D0AE0CF2FC557 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:54,964 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_STARTED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, taskId=task_1463536074763_0001_1_00_000000, scheduledTime=1463536074963, launchTime=1463536074963
2016-05-18 01:47:54,964 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EC20C9688ADF47DAAA35F5BD8FF80D87 in vertex: F243FE12DA264862873D0AE0CF2FC557 numBipartiteSourceTasks: 1
2016-05-18 01:47:54,964 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:54,969 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:54,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536074763_0001_00_000001, launchTime=1463536074970
2016-05-18 01:47:54,971 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074763_0001_00_000001
2016-05-18 01:47:54,972 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074763_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:54,972 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536074763_0001_1_00_000000_0] started. Is using containerId: [container_1463536074763_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:54,972 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0, startTime=1463536074972, containerId=container_1463536074763_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:54 Starting to run new task attempt: attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:54,973 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:54,973 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:54,973 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:54,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false, VertexName: EC20C9688ADF47DAAA35F5BD8FF80D87, VertexParallelism: 1, TaskAttemptID:attempt_1463536074763_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=63C908ACCEF44FE0BC97C79F46C7D3D1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F243FE12DA264862873D0AE0CF2FC557, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:54,973 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:54,975 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536074763_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:54,975 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536074763_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:54,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:54,977 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536074763_0001, with dag id: dag_1463536074763_0001_1
2016-05-18 01:47:54,985 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:54,990 INFO input.MRInput (MRInput.java:initialize(440)) - 63C908ACCEF44FE0BC97C79F46C7D3D1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:54,990 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 63C908ACCEF44FE0BC97C79F46C7D3D1
2016-05-18 01:47:54,995 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EC20C9688ADF47DAAA35F5BD8FF80D87, ordinal: 0
2016-05-18 01:47:54,996 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EC20C9688ADF47DAAA35F5BD8FF80D87, mem on start (mb), free: 293, total: 356, max: 672
2016-05-18 01:47:54,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:54,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:54,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:54,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:54,996 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[63C908ACCEF44FE0BC97C79F46C7D3D1:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F243FE12DA264862873D0AE0CF2FC557:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:54,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:54,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:54,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:54,998 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[63C908ACCEF44FE0BC97C79F46C7D3D1:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F243FE12DA264862873D0AE0CF2FC557:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:54,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:54,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 63C908ACCEF44FE0BC97C79F46C7D3D1
2016-05-18 01:47:54,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 63C908ACCEF44FE0BC97C79F46C7D3D1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:54,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:55,000 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:55,000 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:55,001 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:55,004 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:55,005 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 63C908ACCEF44FE0BC97C79F46C7D3D1 initialized RecordReader from event
2016-05-18 01:47:55,005 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:55,006 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 63C908ACCEF44FE0BC97C79F46C7D3D1
2016-05-18 01:47:55,006 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 1194873E676C4ABFB1BC827C802137D2
2016-05-18 01:47:55,007 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 1194873E676C4ABFB1BC827C802137D2
2016-05-18 01:47:55,008 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:55,008 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:55,011 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F243FE12DA264862873D0AE0CF2FC557 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@32f29a95, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@5582c654, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:55,012 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F243FE12DA264862873D0AE0CF2FC557: , UsingHashComparator=false
2016-05-18 01:47:55,025 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:55,025 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:55,026 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F243FE12DA264862873D0AE0CF2FC557: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:55,026 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F243FE12DA264862873D0AE0CF2FC557: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:55,026 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:55,027 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EC20C9688ADF47DAAA35F5BD8FF80D87, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:55,033 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:55,033 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EC20C9688ADF47DAAA35F5BD8FF80D87, mem on close (mb), free: 191, total: 356, max: 672
2016-05-18 01:47:55,033 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F243FE12DA264862873D0AE0CF2FC557: Starting flush of map output
2016-05-18 01:47:55,033 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F243FE12DA264862873D0AE0CF2FC557: Span0.length = 37, perItem = 24
2016-05-18 01:47:55,035 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F243FE12DA264862873D0AE0CF2FC557: done sorting span=0, length=37, time=1
2016-05-18 01:47:55,035 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F243FE12DA264862873D0AE0CF2FC557: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:55,037 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F243FE12DA264862873D0AE0CF2FC557: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-local-dir/output/attempt_1463536074763_0001_1_00_000000_0_10059_0/file.out
2016-05-18 01:47:55,040 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:55 Completed running task attempt: attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:55,042 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:55,042 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536074763_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:55,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536074763_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536075027, Process_Duration=6, Process_End_Time=1463536075033, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:55,044 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536074763_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,044 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,044 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:55,044 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,044 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, taskAttemptId=attempt_1463536074763_0001_1_00_000000_0, creationTime=1463536074964, allocationTime=1463536074970, startTime=1463536074972, finishTime=1463536075044, timeTaken=72, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:55,045 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074763_0001_00_000001
2016-05-18 01:47:55,045 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536074763_0001_1_00_000000_0
2016-05-18 01:47:55,046 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_FINISHED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, taskId=task_1463536074763_0001_1_00_000000, startTime=1463536074972, finishTime=1463536075045, timeTaken=73, status=SUCCEEDED, successfulAttemptID=attempt_1463536074763_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536075027, Process_Duration=6, Process_End_Time=1463536075033, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:55,046 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:55,046 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:55,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_FINISHED]: vertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, vertexId=vertex_1463536074763_0001_1_00, initRequestedTime=1463536074942, initedTime=1463536074956, startRequestedTime=1463536074944, startedTime=1463536074957, finishTime=1463536075046, timeTaken=89, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536075027, Process_Duration=6, Process_End_Time=1463536075033, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536074972, firstTasksToStart=[ task_1463536074763_0001_1_00_000000 ], lastTaskFinishTime=1463536075044, lastTasksToFinish=[ task_1463536074763_0001_1_00_000000 ], minTaskDuration=72, maxTaskDuration=72, avgTaskDuration=72.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536074763_0001_1_00_000000 ], longestDurationTasks=[ task_1463536074763_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:55,048 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:55,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536074763_0001_00_000001, stoppedTime=1463536075048, exitStatus=0
2016-05-18 01:47:55,048 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] attempt: attempt_1463536074763_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:55,049 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:55,049 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,049 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F243FE12DA264862873D0AE0CF2FC557
2016-05-18 01:47:55,049 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, destinationVertexName=F243FE12DA264862873D0AE0CF2FC557. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:55,049 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:55,050 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536074763_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:55,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_STARTED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, taskId=task_1463536074763_0001_1_01_000000, scheduledTime=1463536075050, launchTime=1463536075050
2016-05-18 01:47:55,050 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:55,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536074763_0001_00_000002, launchTime=1463536075054
2016-05-18 01:47:55,055 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536074763_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:55,055 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074763_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:55,056 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536074763_0001_00_000001, Exiting
2016-05-18 01:47:55,056 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536074763_0001_00_000001
2016-05-18 01:47:55,056 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536074763_0001_00_000001
2016-05-18 01:47:55,055 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074763_0001_00_000002
2016-05-18 01:47:55,058 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536074763_0001_1_01_000000_0] started. Is using containerId: [container_1463536074763_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:55,058 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0, startTime=1463536075058, containerId=container_1463536074763_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:55,059 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536074763_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:55,059 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536074763_0001_00_000001 completed successfully
2016-05-18 01:47:55,060 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074763_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55 Starting to run new task attempt: attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,060 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,060 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,061 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:55,061 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false, VertexName: F243FE12DA264862873D0AE0CF2FC557, VertexParallelism: 1, TaskAttemptID:attempt_1463536074763_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EC20C9688ADF47DAAA35F5BD8FF80D87, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FF605BAF830F479BB43B6C39C96A6300, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:55,061 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:55,062 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,063 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536074763_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:55,063 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536074763_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:55,065 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:55,066 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:55,077 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F243FE12DA264862873D0AE0CF2FC557, ordinal: 1
2016-05-18 01:47:55,077 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F243FE12DA264862873D0AE0CF2FC557, mem on start (mb), free: 185, total: 356, max: 672
2016-05-18 01:47:55,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:55,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:55,078 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:55,079 INFO output.MROutput (MROutput.java:initialize(421)) - FF605BAF830F479BB43B6C39C96A6300: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:55,079 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:55,080 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EC20C9688ADF47DAAA35F5BD8FF80D87:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [FF605BAF830F479BB43B6C39C96A6300:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:55,080 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:55,080 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:55,081 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:55,081 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EC20C9688ADF47DAAA35F5BD8FF80D87:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [FF605BAF830F479BB43B6C39C96A6300:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:55,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:55,082 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EC20C9688ADF47DAAA35F5BD8FF80D87: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:55,082 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EC20C9688ADF47DAAA35F5BD8FF80D87: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:55,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EC20C9688ADF47DAAA35F5BD8FF80D87 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:55,083 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EC20C9688ADF47DAAA35F5BD8FF80D87 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:55,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:55,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:55,084 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EC20C9688ADF47DAAA35F5BD8FF80D87
2016-05-18 01:47:55,086 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:55,086 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,088 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EC20C9688ADF47DAAA35F5BD8FF80D87: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:55,091 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536074763_0001_1_00_000000_0_10059} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536075091, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:47:55,091 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EC20C9688ADF47DAAA35F5BD8FF80D87
2016-05-18 01:47:55,092 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:55,092 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EC20C9688ADF47DAAA35F5BD8FF80D87, wasInterrupted=false
2016-05-18 01:47:55,091 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:55,093 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:55,093 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - EC20C9688ADF47DAAA35F5BD8FF80D87: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:55,094 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:55,094 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 1194873E676C4ABFB1BC827C802137D2
2016-05-18 01:47:55,095 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false"], id: FF605BAF830F479BB43B6C39C96A6300
2016-05-18 01:47:55,095 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false"]
2016-05-18 01:47:55,095 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 1194873E676C4ABFB1BC827C802137D2, for 1 inputs
2016-05-18 01:47:55,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001_wd/localmode-local-dir/output/attempt_1463536074763_0001_1_00_000000_0_10059/file.out, len=732, isLocal=true
2016-05-18 01:47:55,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:55,096 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:55,096 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:55,097 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:55,097 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EC20C9688ADF47DAAA35F5BD8FF80D87
2016-05-18 01:47:55,097 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EC20C9688ADF47DAAA35F5BD8FF80D87: Shuffle Runner thread complete
2016-05-18 01:47:55,097 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F243FE12DA264862873D0AE0CF2FC557, all 1 inputs ready in: 00:00:00.002
2016-05-18 01:47:55,099 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EC20C9688ADF47DAAA35F5BD8FF80D87: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:55,105 INFO output.MROutput (MROutput.java:close(526)) - FF605BAF830F479BB43B6C39C96A6300 closed
2016-05-18 01:47:55,105 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536074763_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:55,106 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,106 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F243FE12DA264862873D0AE0CF2FC557, mem on close (mb), free: 180, total: 356, max: 672
2016-05-18 01:47:55,107 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EC20C9688ADF47DAAA35F5BD8FF80D87
2016-05-18 01:47:55,107 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EC20C9688ADF47DAAA35F5BD8FF80D87: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:55,107 INFO output.MROutput (MROutput.java:close(526)) - FF605BAF830F479BB43B6C39C96A6300 closed
2016-05-18 01:47:55,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:55 Completed running task attempt: attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,108 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:55,109 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536074763_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:55,109 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536074763_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536075097, Process_Duration=9, Process_End_Time=1463536075106, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:55,111 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536074763_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,111 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, taskAttemptId=attempt_1463536074763_0001_1_01_000000_0, creationTime=1463536075050, allocationTime=1463536075053, startTime=1463536075058, finishTime=1463536075111, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:55,111 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536074763_0001_1_01_000000_0
2016-05-18 01:47:55,111 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,112 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:55,112 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,112 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536074763_0001_00_000002
2016-05-18 01:47:55,112 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:TASK_FINISHED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, taskId=task_1463536074763_0001_1_01_000000, startTime=1463536075058, finishTime=1463536075112, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463536074763_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536075097, Process_Duration=9, Process_End_Time=1463536075106, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:55,113 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:55,113 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557]
2016-05-18 01:47:55,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:VERTEX_FINISHED]: vertexName=F243FE12DA264862873D0AE0CF2FC557, vertexId=vertex_1463536074763_0001_1_01, initRequestedTime=1463536074944, initedTime=1463536074949, startRequestedTime=1463536074958, startedTime=1463536074959, finishTime=1463536075113, timeTaken=154, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536075097, Process_Duration=9, Process_End_Time=1463536075106, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536075058, firstTasksToStart=[ task_1463536074763_0001_1_01_000000 ], lastTaskFinishTime=1463536075111, lastTasksToFinish=[ task_1463536074763_0001_1_01_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536074763_0001_1_01_000000 ], longestDurationTasks=[ task_1463536074763_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:55,114 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:55,115 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536074763_0001_00_000002, stoppedTime=1463536075115, exitStatus=0
2016-05-18 01:47:55,115 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:55,115 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,115 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536074763_0001_1_00 [EC20C9688ADF47DAAA35F5BD8FF80D87]
2016-05-18 01:47:55,115 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536074763_0001_1
2016-05-18 01:47:55,115 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536074763_0001_1
2016-05-18 01:47:55,115 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074763_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:55,116 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: FF605BAF830F479BB43B6C39C96A6300 for vertex: vertex_1463536074763_0001_1_01 [F243FE12DA264862873D0AE0CF2FC557], outputName: FF605BAF830F479BB43B6C39C96A6300
2016-05-18 01:47:55,116 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536074763_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:55,123 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536074763_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:55,123 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536074763_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:55,123 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536074763_0001_00_000002, Exiting
2016-05-18 01:47:55,123 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536074763_0001_00_000002
2016-05-18 01:47:55,123 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536074763_0001_00_000002
2016-05-18 01:47:55,123 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536074763_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:55,124 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536074763_0001_00_000002 completed successfully
2016-05-18 01:47:55,126 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:FF605BAF830F479BB43B6C39C96A6300 of vertex/vertexGroup:F243FE12DA264862873D0AE0CF2FC557 isVertexGroupOutput:false
2016-05-18 01:47:55,126 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,129 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536074763_0001_1, queueSize=0
2016-05-18 01:47:55,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536074763_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536074763_0001_1, startTime=1463536074941, finishTime=1463536075127, timeTaken=186, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=430, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072150124, Process_Duration=15, Process_End_Time=2927072150139, Read_Duration=6, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:55,130 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536074763_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:55,131 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536074763_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:55 Completed Dag: dag_1463536074763_0001_1
2016-05-18 01:47:55,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536074763_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:55,132 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536074763_0001_1_post
2016-05-18 01:47:55,132 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:55,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:55,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false, with id=dag_1463536074763_0001_1
2016-05-18 01:47:55,133 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D] comprehensivesort/(1/1) ...ive/true_true_false_false, with id=dag_1463536074763_0001_1
2016-05-18 01:47:55,137 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/true_true_false_false, applicationId=application_1463536074763_0001
2016-05-18 01:47:55,140 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:55,140 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:55,140 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:55,140 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:55,140 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:55,141 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:55,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:55,143 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:55,143 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:55,143 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:55,144 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:55,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:55,145 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:55,146 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:55,149 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/2E7CD4AE09A046EEAF21F37803A4A9DC/F3ED5E42968143B2959A3B8F909B2F9D/.tez/application_1463536074763_0001
2016-05-18 01:47:55,150 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false/_temporary
2016-05-18 01:47:55,150 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.409
2016-05-18 01:47:55,224 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:55,224 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:55,268 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 01:47:55,278 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.025
2016-05-18 01:47:55,278 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:55,278 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:55,278 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:55,295 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:55,302 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false
2016-05-18 01:47:55,356 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:55,356 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:55,356 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false"]
2016-05-18 01:47:55,356 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:55,357 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:55,357 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:55,359 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ve/false_true_false_false
2016-05-18 01:47:55,360 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:55,384 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:55,389 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:55,389 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:55,389 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:55,400 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001 doesn't exist and is created
2016-05-18 01:47:55,429 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:55,429 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:55,430 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd
2016-05-18 01:47:55,448 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536075389_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:55,448 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:55,449 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:55,451 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:55,451 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:55,451 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:55,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:55,453 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:55,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:55,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:55,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:55,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:55,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:55,456 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:55,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:55,457 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:55,457 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:55,457 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:55,458 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:55,458 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:55,460 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:55,461 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:55,461 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:55,461 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-log-dir
2016-05-18 01:47:55,462 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-log-dir/history.txt.appattempt_1463536075389_0001_000000, maxErrors=10
2016-05-18 01:47:55,462 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:55,462 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536075389_0001_000000, appSubmitTime=1463536075447, launchTime=1463536075447
2016-05-18 01:47:55,467 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:55,475 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:35096
2016-05-18 01:47:55,476 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536075389
2016-05-18 01:47:55,477 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536075389_0001_000000, startTime=1463536075477
2016-05-18 01:47:55,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:55,530 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:55,530 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:55,530 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ve/false_true_false_false, applicationId=application_1463536075389_0001, dagName=[A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false
2016-05-18 01:47:55,549 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false
2016-05-18 01:47:55,549 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:55,550 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536075389_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-log-dir/dag_1463536075389_0001_1.dot
2016-05-18 01:47:55,551 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-log-dir/dag_1463536075389_0001_1-tez-dag.pb.txt
2016-05-18 01:47:55,557 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536075389_0001_1
2016-05-18 01:47:55,557 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,558 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false
2016-05-18 01:47:55 Running Dag: dag_1463536075389_0001_1
2016-05-18 01:47:55,573 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536075389_0001_1, submitTime=1463536075549
2016-05-18 01:47:55,574 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001/tez-conf.pb" } size: 40937 timestamp: 1463536075000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:55,574 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,575 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:55,576 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001/tez-conf.pb" } size: 40937 timestamp: 1463536075000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:55,576 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,577 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:55,577 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:55,578 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536075389_0001_1, initTime=1463536075577
2016-05-18 01:47:55,578 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536075389_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:55,579 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:55,579 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:DAG_STARTED]: dagID=dag_1463536075389_0001_1, startTime=1463536075579
2016-05-18 01:47:55,579 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536075389_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:55,579 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A1F1442BAECA4240B6F65DDE55166C06 : {6895B8EAD00946AA9C9B5EA6C1C78BC6={InputName=6895B8EAD00946AA9C9B5EA6C1C78BC6}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:55,580 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 6895B8EAD00946AA9C9B5EA6C1C78BC6, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:55,580 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ve/false_true_false_false, applicationId=application_1463536075389_0001, dagName=[A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false
2016-05-18 01:47:55,580 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,580 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,580 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,580 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,581 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:55,581 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 6895B8EAD00946AA9C9B5EA6C1C78BC6 on vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,581 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,583 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 6895B8EAD00946AA9C9B5EA6C1C78BC6 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:55,586 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:55,586 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,586 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:55,586 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,587 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:55,587 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:55,587 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536075389_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:55,588 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5], numAdditionalOutputs=1
2016-05-18 01:47:55,588 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=E984A5E9C5B8437A9B2F664D2E60276E, vertex=vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:55,588 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:55,589 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 6895B8EAD00946AA9C9B5EA6C1C78BC6 on vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,591 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FECF64452EEF42EB9BD9588137B5D0F5:E984A5E9C5B8437A9B2F664D2E60276E using oldmapred API
2016-05-18 01:47:55,591 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=E984A5E9C5B8437A9B2F664D2E60276E, vertexName=FECF64452EEF42EB9BD9588137B5D0F5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:55,592 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, vertexId=vertex_1463536075389_0001_1_01, initRequestedTime=1463536075581, initedTime=1463536075588, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:55,593 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:55,594 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:55,594 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {6895B8EAD00946AA9C9B5EA6C1C78BC6=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:55,596 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] parallelism set to 1
2016-05-18 01:47:55,596 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:55,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, vertexId=vertex_1463536075389_0001_1_00, initRequestedTime=1463536075579, initedTime=1463536075597, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:55,597 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] with distanceFromRoot: 0
2016-05-18 01:47:55,598 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:55,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536075389_0001_1_00, startRequestedTime=1463536075581, startedTime=1463536075598
2016-05-18 01:47:55,599 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536075389_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:55,599 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:55,600 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536075389_0001_1_00 for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] numStartedSources: 1 numSources: 1
2016-05-18 01:47:55,600 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] with distanceFromRoot: 1
2016-05-18 01:47:55,600 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A1F1442BAECA4240B6F65DDE55166C06
2016-05-18 01:47:55,601 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,604 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536075389_0001, with dag id: dag_1463536075389_0001_1
2016-05-18 01:47:55,602 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536075389_0001_1_01, startRequestedTime=1463536075600, startedTime=1463536075600
2016-05-18 01:47:55,605 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:55,606 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FECF64452EEF42EB9BD9588137B5D0F5 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:55,606 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_STARTED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, taskId=task_1463536075389_0001_1_00_000000, scheduledTime=1463536075606, launchTime=1463536075606
2016-05-18 01:47:55,607 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,607 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A1F1442BAECA4240B6F65DDE55166C06 in vertex: FECF64452EEF42EB9BD9588137B5D0F5 numBipartiteSourceTasks: 1
2016-05-18 01:47:55,608 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:55,612 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536075389_0001_00_000001, launchTime=1463536075611
2016-05-18 01:47:55,612 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536075389_0001_00_000001
2016-05-18 01:47:55,613 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536075389_0001_1_00_000000_0] started. Is using containerId: [container_1463536075389_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:55,613 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0, startTime=1463536075613, containerId=container_1463536075389_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:55,614 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536075389_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55 Starting to run new task attempt: attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,614 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,614 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,614 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:55,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false, VertexName: A1F1442BAECA4240B6F65DDE55166C06, VertexParallelism: 1, TaskAttemptID:attempt_1463536075389_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=6895B8EAD00946AA9C9B5EA6C1C78BC6, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FECF64452EEF42EB9BD9588137B5D0F5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:55,615 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:55,615 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,615 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536075389_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:55,616 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536075389_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:55,619 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:55,620 INFO input.MRInput (MRInput.java:initialize(440)) - 6895B8EAD00946AA9C9B5EA6C1C78BC6 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:55,620 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 6895B8EAD00946AA9C9B5EA6C1C78BC6
2016-05-18 01:47:55,625 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A1F1442BAECA4240B6F65DDE55166C06, ordinal: 0
2016-05-18 01:47:55,625 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A1F1442BAECA4240B6F65DDE55166C06, mem on start (mb), free: 280, total: 347, max: 672
2016-05-18 01:47:55,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:55,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:55,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:55,625 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:55,625 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6895B8EAD00946AA9C9B5EA6C1C78BC6:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FECF64452EEF42EB9BD9588137B5D0F5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:55,625 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:55,625 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:55,626 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:55,626 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6895B8EAD00946AA9C9B5EA6C1C78BC6:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FECF64452EEF42EB9BD9588137B5D0F5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:55,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:55,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 6895B8EAD00946AA9C9B5EA6C1C78BC6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:55,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:55,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:55,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 6895B8EAD00946AA9C9B5EA6C1C78BC6
2016-05-18 01:47:55,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:55,628 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,631 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:55,631 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 6895B8EAD00946AA9C9B5EA6C1C78BC6 initialized RecordReader from event
2016-05-18 01:47:55,631 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:55,632 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 6895B8EAD00946AA9C9B5EA6C1C78BC6
2016-05-18 01:47:55,633 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 718FFB2BF5244CE1AF554D5D6DC09C1F
2016-05-18 01:47:55,633 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 718FFB2BF5244CE1AF554D5D6DC09C1F
2016-05-18 01:47:55,637 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FECF64452EEF42EB9BD9588137B5D0F5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2241ba0b, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@91d39bf, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:55,637 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FECF64452EEF42EB9BD9588137B5D0F5: , UsingHashComparator=false
2016-05-18 01:47:55,651 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:55,651 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:55,652 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FECF64452EEF42EB9BD9588137B5D0F5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:55,652 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FECF64452EEF42EB9BD9588137B5D0F5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:55,652 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:55,652 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A1F1442BAECA4240B6F65DDE55166C06, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:55,659 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,659 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A1F1442BAECA4240B6F65DDE55166C06, mem on close (mb), free: 178, total: 347, max: 672
2016-05-18 01:47:55,660 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FECF64452EEF42EB9BD9588137B5D0F5: Starting flush of map output
2016-05-18 01:47:55,660 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FECF64452EEF42EB9BD9588137B5D0F5: Span0.length = 37, perItem = 24
2016-05-18 01:47:55,662 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FECF64452EEF42EB9BD9588137B5D0F5: done sorting span=0, length=37, time=2
2016-05-18 01:47:55,663 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FECF64452EEF42EB9BD9588137B5D0F5: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:55,664 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FECF64452EEF42EB9BD9588137B5D0F5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-local-dir/output/attempt_1463536075389_0001_1_00_000000_0_10064_0/file.out
2016-05-18 01:47:55,667 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:55 Completed running task attempt: attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,668 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:55,668 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536075389_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:55,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536075389_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536075652, Process_Duration=7, Process_End_Time=1463536075659, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:55,669 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536075389_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,669 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,669 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:55,669 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, taskAttemptId=attempt_1463536075389_0001_1_00_000000_0, creationTime=1463536075606, allocationTime=1463536075611, startTime=1463536075613, finishTime=1463536075669, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:55,670 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536075389_0001_1_00_000000_0
2016-05-18 01:47:55,670 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,670 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536075389_0001_00_000001
2016-05-18 01:47:55,670 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_FINISHED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, taskId=task_1463536075389_0001_1_00_000000, startTime=1463536075613, finishTime=1463536075670, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463536075389_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536075652, Process_Duration=7, Process_End_Time=1463536075659, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:55,671 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:55,671 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,673 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_FINISHED]: vertexName=A1F1442BAECA4240B6F65DDE55166C06, vertexId=vertex_1463536075389_0001_1_00, initRequestedTime=1463536075579, initedTime=1463536075597, startRequestedTime=1463536075581, startedTime=1463536075598, finishTime=1463536075671, timeTaken=73, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536075652, Process_Duration=7, Process_End_Time=1463536075659, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536075613, firstTasksToStart=[ task_1463536075389_0001_1_00_000000 ], lastTaskFinishTime=1463536075669, lastTasksToFinish=[ task_1463536075389_0001_1_00_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536075389_0001_1_00_000000 ], longestDurationTasks=[ task_1463536075389_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:55,673 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:55,674 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] attempt: attempt_1463536075389_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:55,674 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FECF64452EEF42EB9BD9588137B5D0F5
2016-05-18 01:47:55,675 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536075389_0001_00_000001, stoppedTime=1463536075674, exitStatus=0
2016-05-18 01:47:55,675 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:55,675 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,675 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A1F1442BAECA4240B6F65DDE55166C06, destinationVertexName=FECF64452EEF42EB9BD9588137B5D0F5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:55,675 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,675 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536075389_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:55,676 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_STARTED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, taskId=task_1463536075389_0001_1_01_000000, scheduledTime=1463536075676, launchTime=1463536075676
2016-05-18 01:47:55,677 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,678 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536075389_0001_00_000002, launchTime=1463536075678
2016-05-18 01:47:55,679 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536075389_0001_00_000002
2016-05-18 01:47:55,679 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536075389_0001_1_01_000000_0] started. Is using containerId: [container_1463536075389_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:55,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0, startTime=1463536075679, containerId=container_1463536075389_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:55,680 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536075389_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55 Starting to run new task attempt: attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,681 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,681 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,681 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:55,681 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false, VertexName: FECF64452EEF42EB9BD9588137B5D0F5, VertexParallelism: 1, TaskAttemptID:attempt_1463536075389_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A1F1442BAECA4240B6F65DDE55166C06, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E984A5E9C5B8437A9B2F664D2E60276E, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:55,682 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:55,682 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,683 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536075389_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:55,683 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536075389_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:55,683 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536075389_0001_00_000001, Exiting
2016-05-18 01:47:55,683 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536075389_0001_00_000001
2016-05-18 01:47:55,683 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536075389_0001_00_000001
2016-05-18 01:47:55,683 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536075389_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,684 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536075389_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:55,687 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:55,692 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536075389_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:55,692 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536075389_0001_00_000001 completed successfully
2016-05-18 01:47:55,693 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:55,695 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FECF64452EEF42EB9BD9588137B5D0F5, ordinal: 1
2016-05-18 01:47:55,695 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FECF64452EEF42EB9BD9588137B5D0F5, mem on start (mb), free: 172, total: 347, max: 672
2016-05-18 01:47:55,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:55,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:55,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:55,702 INFO output.MROutput (MROutput.java:initialize(421)) - E984A5E9C5B8437A9B2F664D2E60276E: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:55,703 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:55,703 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E984A5E9C5B8437A9B2F664D2E60276E:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A1F1442BAECA4240B6F65DDE55166C06:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:55,703 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:55,703 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:55,703 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:55,704 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E984A5E9C5B8437A9B2F664D2E60276E:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A1F1442BAECA4240B6F65DDE55166C06:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:55,704 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:55,704 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A1F1442BAECA4240B6F65DDE55166C06 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:55,704 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:55,704 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:55,704 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A1F1442BAECA4240B6F65DDE55166C06: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:55,704 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A1F1442BAECA4240B6F65DDE55166C06: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:55,706 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A1F1442BAECA4240B6F65DDE55166C06 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:55,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A1F1442BAECA4240B6F65DDE55166C06
2016-05-18 01:47:55,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:55,707 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,708 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A1F1442BAECA4240B6F65DDE55166C06: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:55,710 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536075389_0001_1_00_000000_0_10064} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536075710, TimeTaken=2, Rate=0.35 MB/s
2016-05-18 01:47:55,710 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A1F1442BAECA4240B6F65DDE55166C06
2016-05-18 01:47:55,711 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:55,711 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:55,711 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A1F1442BAECA4240B6F65DDE55166C06, wasInterrupted=false
2016-05-18 01:47:55,711 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 718FFB2BF5244CE1AF554D5D6DC09C1F
2016-05-18 01:47:55,711 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:55,712 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false"], id: E984A5E9C5B8437A9B2F664D2E60276E
2016-05-18 01:47:55,713 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false"]
2016-05-18 01:47:55,713 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 718FFB2BF5244CE1AF554D5D6DC09C1F, for 1 inputs
2016-05-18 01:47:55,713 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A1F1442BAECA4240B6F65DDE55166C06: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:55,713 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:55,715 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001_wd/localmode-local-dir/output/attempt_1463536075389_0001_1_00_000000_0_10064/file.out, len=732, isLocal=true
2016-05-18 01:47:55,715 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:55,715 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:55,715 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:55,716 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:55,716 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A1F1442BAECA4240B6F65DDE55166C06
2016-05-18 01:47:55,716 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A1F1442BAECA4240B6F65DDE55166C06: Shuffle Runner thread complete
2016-05-18 01:47:55,716 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FECF64452EEF42EB9BD9588137B5D0F5, all 1 inputs ready in: 00:00:00.002
2016-05-18 01:47:55,718 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A1F1442BAECA4240B6F65DDE55166C06: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:55,725 INFO output.MROutput (MROutput.java:close(526)) - E984A5E9C5B8437A9B2F664D2E60276E closed
2016-05-18 01:47:55,725 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536075389_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:55,726 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,727 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FECF64452EEF42EB9BD9588137B5D0F5, mem on close (mb), free: 168, total: 347, max: 672
2016-05-18 01:47:55,727 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A1F1442BAECA4240B6F65DDE55166C06
2016-05-18 01:47:55,727 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A1F1442BAECA4240B6F65DDE55166C06: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:55,727 INFO output.MROutput (MROutput.java:close(526)) - E984A5E9C5B8437A9B2F664D2E60276E closed
2016-05-18 01:47:55,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:55 Completed running task attempt: attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,728 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:55,729 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536075389_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:55,730 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536075389_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536075717, Process_Duration=9, Process_End_Time=1463536075726, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:55,731 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536075389_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,731 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:55,731 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, taskAttemptId=attempt_1463536075389_0001_1_01_000000_0, creationTime=1463536075676, allocationTime=1463536075677, startTime=1463536075679, finishTime=1463536075731, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:55,731 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:55,732 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,732 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536075389_0001_1_01_000000_0
2016-05-18 01:47:55,732 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536075389_0001_00_000002
2016-05-18 01:47:55,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:TASK_FINISHED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, taskId=task_1463536075389_0001_1_01_000000, startTime=1463536075679, finishTime=1463536075732, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463536075389_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536075717, Process_Duration=9, Process_End_Time=1463536075726, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:55,733 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:55,733 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5]
2016-05-18 01:47:55,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:VERTEX_FINISHED]: vertexName=FECF64452EEF42EB9BD9588137B5D0F5, vertexId=vertex_1463536075389_0001_1_01, initRequestedTime=1463536075581, initedTime=1463536075588, startRequestedTime=1463536075600, startedTime=1463536075600, finishTime=1463536075733, timeTaken=133, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536075717, Process_Duration=9, Process_End_Time=1463536075726, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536075679, firstTasksToStart=[ task_1463536075389_0001_1_01_000000 ], lastTaskFinishTime=1463536075731, lastTasksToFinish=[ task_1463536075389_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536075389_0001_1_01_000000 ], longestDurationTasks=[ task_1463536075389_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:55,735 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:55,735 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536075389_0001_00_000002, stoppedTime=1463536075735, exitStatus=0
2016-05-18 01:47:55,735 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:55,735 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,736 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536075389_0001_1_00 [A1F1442BAECA4240B6F65DDE55166C06]
2016-05-18 01:47:55,736 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536075389_0001_1
2016-05-18 01:47:55,736 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536075389_0001_1
2016-05-18 01:47:55,736 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536075389_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:55,736 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: E984A5E9C5B8437A9B2F664D2E60276E for vertex: vertex_1463536075389_0001_1_01 [FECF64452EEF42EB9BD9588137B5D0F5], outputName: E984A5E9C5B8437A9B2F664D2E60276E
2016-05-18 01:47:55,740 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536075389_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:55,743 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536075389_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:55,743 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536075389_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:55,743 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536075389_0001_00_000002, Exiting
2016-05-18 01:47:55,744 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536075389_0001_00_000002
2016-05-18 01:47:55,744 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536075389_0001_00_000002
2016-05-18 01:47:55,744 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536075389_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:55,744 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536075389_0001_00_000002 completed successfully
2016-05-18 01:47:55,748 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:E984A5E9C5B8437A9B2F664D2E60276E of vertex/vertexGroup:FECF64452EEF42EB9BD9588137B5D0F5 isVertexGroupOutput:false
2016-05-18 01:47:55,749 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:55,751 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536075389_0001_1, queueSize=0
2016-05-18 01:47:55,752 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536075389_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536075389_0001_1, startTime=1463536075579, finishTime=1463536075749, timeTaken=170, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=360, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072151369, Process_Duration=16, Process_End_Time=2927072151385, Read_Duration=5, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:55,752 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536075389_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:55,752 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536075389_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:55 Completed Dag: dag_1463536075389_0001_1
2016-05-18 01:47:55,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536075389_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:55,753 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536075389_0001_1_post
2016-05-18 01:47:55,753 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:55,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:55,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:55,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false, with id=dag_1463536075389_0001_1
2016-05-18 01:47:55,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D] comprehensivesort/(1/1) ...ve/false_true_false_false, with id=dag_1463536075389_0001_1
2016-05-18 01:47:55,759 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ve/false_true_false_false, applicationId=application_1463536075389_0001
2016-05-18 01:47:55,761 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:55,761 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:55,761 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:55,761 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:55,761 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:55,762 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:55,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:55,762 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:55,762 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:55,763 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:55,763 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:55,763 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:55,763 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:55,764 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:55,769 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A42A37FD191847D5A0863A55BE3297E9/E1BFCCBCD94A47E680F0CC2204D6CD0D/.tez/application_1463536075389_0001
2016-05-18 01:47:55,769 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false/_temporary
2016-05-18 01:47:55,770 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.413
2016-05-18 01:47:55,845 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:55,845 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:55,883 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 01:47:55,893 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 01:47:55,894 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:55,894 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:55,894 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:55,920 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:55,932 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false
2016-05-18 01:47:56,015 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:56,016 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,016 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false"]
2016-05-18 01:47:56,016 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:56,017 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:56,017 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:56,018 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ve/false_false_true_false
2016-05-18 01:47:56,018 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:56,041 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:56,045 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:56,045 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:56,046 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:56,055 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001 doesn't exist and is created
2016-05-18 01:47:56,075 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:56,076 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:56,076 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd
2016-05-18 01:47:56,092 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536076045_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:56,092 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:56,092 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:56,093 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:56,093 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:56,093 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:56,094 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:56,094 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:56,095 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:56,095 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:56,096 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:56,096 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:56,096 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:56,098 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:56,098 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:56,098 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:56,098 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-log-dir
2016-05-18 01:47:56,099 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-log-dir/history.txt.appattempt_1463536076045_0001_000000, maxErrors=10
2016-05-18 01:47:56,099 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:56,099 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536076045_0001_000000, appSubmitTime=1463536076091, launchTime=1463536076091
2016-05-18 01:47:56,103 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:56,106 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:49809
2016-05-18 01:47:56,106 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536076045
2016-05-18 01:47:56,108 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536076045_0001_000000, startTime=1463536076108
2016-05-18 01:47:56,108 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:56,176 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:56,176 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:56,176 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ve/false_false_true_false, applicationId=application_1463536076045_0001, dagName=[8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false
2016-05-18 01:47:56,195 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false
2016-05-18 01:47:56,196 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:56,196 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536076045_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-log-dir/dag_1463536076045_0001_1.dot
2016-05-18 01:47:56,197 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-log-dir/dag_1463536076045_0001_1-tez-dag.pb.txt
2016-05-18 01:47:56,203 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536076045_0001_1
2016-05-18 01:47:56,203 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,203 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false
2016-05-18 01:47:56 Running Dag: dag_1463536076045_0001_1
2016-05-18 01:47:56,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536076045_0001_1, submitTime=1463536076196
2016-05-18 01:47:56,218 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001/tez-conf.pb" } size: 40937 timestamp: 1463536076000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:56,218 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,218 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:56,220 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001/tez-conf.pb" } size: 40937 timestamp: 1463536076000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:56,220 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,220 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:56,220 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:56,221 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536076045_0001_1, initTime=1463536076220
2016-05-18 01:47:56,222 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076045_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:56,223 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:56,223 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:DAG_STARTED]: dagID=dag_1463536076045_0001_1, startTime=1463536076223
2016-05-18 01:47:56,223 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076045_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:56,223 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D1B666A065A948A5803476FE1B5CA44B : {A7D9C4B85F014A83B5A180712ED23943={InputName=A7D9C4B85F014A83B5A180712ED23943}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:56,223 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A7D9C4B85F014A83B5A180712ED23943, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:56,223 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,223 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,223 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,224 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,224 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:56,224 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,225 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ve/false_false_true_false, applicationId=application_1463536076045_0001, dagName=[8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false
2016-05-18 01:47:56,225 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A7D9C4B85F014A83B5A180712ED23943 on vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,228 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A7D9C4B85F014A83B5A180712ED23943 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:56,228 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:56,228 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,229 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536076045_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:56,230 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B], numAdditionalOutputs=1
2016-05-18 01:47:56,230 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=2C51194EB45B4C67AFD81BE1A7F7A957, vertex=vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:56,231 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:56,231 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:56,232 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:56,232 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:56,233 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A7D9C4B85F014A83B5A180712ED23943 on vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,233 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C3426BED15574EFAAD55210CE2C8BF0B:2C51194EB45B4C67AFD81BE1A7F7A957 using oldmapred API
2016-05-18 01:47:56,233 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=2C51194EB45B4C67AFD81BE1A7F7A957, vertexName=C3426BED15574EFAAD55210CE2C8BF0B, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:56,234 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, vertexId=vertex_1463536076045_0001_1_01, initRequestedTime=1463536076224, initedTime=1463536076230, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:56,235 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:56,236 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:56,237 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A7D9C4B85F014A83B5A180712ED23943=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:56,237 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] parallelism set to 1
2016-05-18 01:47:56,237 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:56,238 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, vertexId=vertex_1463536076045_0001_1_00, initRequestedTime=1463536076223, initedTime=1463536076237, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:56,238 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] with distanceFromRoot: 0
2016-05-18 01:47:56,238 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:56,239 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536076045_0001_1_00, startRequestedTime=1463536076224, startedTime=1463536076239
2016-05-18 01:47:56,239 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536076045_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:56,240 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:56,240 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536076045_0001_1_00 for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] numStartedSources: 1 numSources: 1
2016-05-18 01:47:56,240 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] with distanceFromRoot: 1
2016-05-18 01:47:56,241 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D1B666A065A948A5803476FE1B5CA44B
2016-05-18 01:47:56,241 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,241 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536076045_0001_1_01, startRequestedTime=1463536076240, startedTime=1463536076241
2016-05-18 01:47:56,242 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:56,243 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C3426BED15574EFAAD55210CE2C8BF0B with 0 source tasks and 1 pending tasks
2016-05-18 01:47:56,243 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D1B666A065A948A5803476FE1B5CA44B in vertex: C3426BED15574EFAAD55210CE2C8BF0B numBipartiteSourceTasks: 1
2016-05-18 01:47:56,244 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_STARTED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, taskId=task_1463536076045_0001_1_00_000000, scheduledTime=1463536076243, launchTime=1463536076243
2016-05-18 01:47:56,244 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,245 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:56,247 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536076045_0001_00_000001, launchTime=1463536076247
2016-05-18 01:47:56,247 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076045_0001_00_000001
2016-05-18 01:47:56,248 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076045_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,248 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536076045_0001_1_00_000000_0] started. Is using containerId: [container_1463536076045_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:56,248 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0, startTime=1463536076248, containerId=container_1463536076045_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:56 Starting to run new task attempt: attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,249 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536076045_0001, with dag id: dag_1463536076045_0001_1
2016-05-18 01:47:56,249 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,251 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,251 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:56,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false, VertexName: D1B666A065A948A5803476FE1B5CA44B, VertexParallelism: 1, TaskAttemptID:attempt_1463536076045_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7D9C4B85F014A83B5A180712ED23943, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C3426BED15574EFAAD55210CE2C8BF0B, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:56,251 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:56,253 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,256 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536076045_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:56,257 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536076045_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:56,258 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:56,259 INFO input.MRInput (MRInput.java:initialize(440)) - A7D9C4B85F014A83B5A180712ED23943 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:56,260 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A7D9C4B85F014A83B5A180712ED23943
2016-05-18 01:47:56,264 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D1B666A065A948A5803476FE1B5CA44B, ordinal: 0
2016-05-18 01:47:56,264 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D1B666A065A948A5803476FE1B5CA44B, mem on start (mb), free: 284, total: 356, max: 672
2016-05-18 01:47:56,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:56,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:56,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:56,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:56,265 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A7D9C4B85F014A83B5A180712ED23943:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C3426BED15574EFAAD55210CE2C8BF0B:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:56,265 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:56,265 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:56,266 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:56,266 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A7D9C4B85F014A83B5A180712ED23943:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C3426BED15574EFAAD55210CE2C8BF0B:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:56,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:56,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A7D9C4B85F014A83B5A180712ED23943 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:56,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:56,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:56,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A7D9C4B85F014A83B5A180712ED23943
2016-05-18 01:47:56,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:56,268 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,270 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:56,270 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A7D9C4B85F014A83B5A180712ED23943 initialized RecordReader from event
2016-05-18 01:47:56,272 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,273 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: A7D9C4B85F014A83B5A180712ED23943
2016-05-18 01:47:56,273 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 44BC9F3A3D1042A0AF558D757E68F816
2016-05-18 01:47:56,273 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 44BC9F3A3D1042A0AF558D757E68F816
2016-05-18 01:47:56,275 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C3426BED15574EFAAD55210CE2C8BF0B using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@18fceccf, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@51f16da, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:56,276 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C3426BED15574EFAAD55210CE2C8BF0B: , UsingHashComparator=false
2016-05-18 01:47:56,279 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:56,280 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:56,288 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:56,288 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:56,289 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C3426BED15574EFAAD55210CE2C8BF0B: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:56,289 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C3426BED15574EFAAD55210CE2C8BF0B: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:56,289 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,289 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D1B666A065A948A5803476FE1B5CA44B, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:56,294 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,294 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D1B666A065A948A5803476FE1B5CA44B, mem on close (mb), free: 182, total: 356, max: 672
2016-05-18 01:47:56,295 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C3426BED15574EFAAD55210CE2C8BF0B: Starting flush of map output
2016-05-18 01:47:56,295 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C3426BED15574EFAAD55210CE2C8BF0B: Span0.length = 37, perItem = 24
2016-05-18 01:47:56,296 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C3426BED15574EFAAD55210CE2C8BF0B: done sorting span=0, length=37, time=1
2016-05-18 01:47:56,296 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C3426BED15574EFAAD55210CE2C8BF0B: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:56,297 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C3426BED15574EFAAD55210CE2C8BF0B: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-local-dir/output/attempt_1463536076045_0001_1_00_000000_0_10070_0/file.out
2016-05-18 01:47:56,300 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:56 Completed running task attempt: attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,301 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:56,301 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536076045_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:56,302 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536076045_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536076289, Process_Duration=5, Process_End_Time=1463536076294, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:56,303 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536076045_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,303 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,303 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:56,303 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,303 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, taskAttemptId=attempt_1463536076045_0001_1_00_000000_0, creationTime=1463536076244, allocationTime=1463536076246, startTime=1463536076248, finishTime=1463536076303, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:56,304 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536076045_0001_1_00_000000_0
2016-05-18 01:47:56,304 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076045_0001_00_000001
2016-05-18 01:47:56,306 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_FINISHED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, taskId=task_1463536076045_0001_1_00_000000, startTime=1463536076248, finishTime=1463536076304, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463536076045_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536076289, Process_Duration=5, Process_End_Time=1463536076294, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:56,306 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:56,306 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,307 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_FINISHED]: vertexName=D1B666A065A948A5803476FE1B5CA44B, vertexId=vertex_1463536076045_0001_1_00, initRequestedTime=1463536076223, initedTime=1463536076237, startRequestedTime=1463536076224, startedTime=1463536076239, finishTime=1463536076306, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536076289, Process_Duration=5, Process_End_Time=1463536076294, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536076248, firstTasksToStart=[ task_1463536076045_0001_1_00_000000 ], lastTaskFinishTime=1463536076303, lastTasksToFinish=[ task_1463536076045_0001_1_00_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536076045_0001_1_00_000000 ], longestDurationTasks=[ task_1463536076045_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:56,307 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:56,307 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536076045_0001_00_000001, stoppedTime=1463536076307, exitStatus=0
2016-05-18 01:47:56,308 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] attempt: attempt_1463536076045_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:56,308 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:56,308 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C3426BED15574EFAAD55210CE2C8BF0B
2016-05-18 01:47:56,308 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D1B666A065A948A5803476FE1B5CA44B, destinationVertexName=C3426BED15574EFAAD55210CE2C8BF0B. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:56,308 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,309 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:56,309 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536076045_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:56,310 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_STARTED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, taskId=task_1463536076045_0001_1_01_000000, scheduledTime=1463536076310, launchTime=1463536076310
2016-05-18 01:47:56,310 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536076045_0001_00_000002, launchTime=1463536076313
2016-05-18 01:47:56,314 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076045_0001_00_000002
2016-05-18 01:47:56,315 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536076045_0001_1_01_000000_0] started. Is using containerId: [container_1463536076045_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:56,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0, startTime=1463536076315, containerId=container_1463536076045_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:56,315 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536076045_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:56,315 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076045_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:56,316 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536076045_0001_00_000001, Exiting
2016-05-18 01:47:56,316 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536076045_0001_00_000001
2016-05-18 01:47:56,316 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536076045_0001_00_000001
2016-05-18 01:47:56,317 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076045_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56 Starting to run new task attempt: attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,318 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,318 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,318 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:56,318 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false, VertexName: C3426BED15574EFAAD55210CE2C8BF0B, VertexParallelism: 1, TaskAttemptID:attempt_1463536076045_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D1B666A065A948A5803476FE1B5CA44B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=2C51194EB45B4C67AFD81BE1A7F7A957, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:56,319 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:56,320 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536076045_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:56,320 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536076045_0001_00_000001 completed successfully
2016-05-18 01:47:56,321 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,322 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536076045_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,322 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536076045_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:56,325 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:56,329 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:56,340 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C3426BED15574EFAAD55210CE2C8BF0B, ordinal: 1
2016-05-18 01:47:56,341 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C3426BED15574EFAAD55210CE2C8BF0B, mem on start (mb), free: 176, total: 356, max: 672
2016-05-18 01:47:56,341 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:56,341 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:56,341 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:56,347 INFO output.MROutput (MROutput.java:initialize(421)) - 2C51194EB45B4C67AFD81BE1A7F7A957: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:56,347 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:56,347 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D1B666A065A948A5803476FE1B5CA44B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [2C51194EB45B4C67AFD81BE1A7F7A957:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:56,348 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:56,348 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:56,348 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:56,348 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D1B666A065A948A5803476FE1B5CA44B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [2C51194EB45B4C67AFD81BE1A7F7A957:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:56,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:56,349 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D1B666A065A948A5803476FE1B5CA44B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:56,350 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:56,350 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:56,351 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D1B666A065A948A5803476FE1B5CA44B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:56,351 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D1B666A065A948A5803476FE1B5CA44B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:56,353 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D1B666A065A948A5803476FE1B5CA44B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:56,356 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D1B666A065A948A5803476FE1B5CA44B
2016-05-18 01:47:56,356 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:56,358 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,359 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D1B666A065A948A5803476FE1B5CA44B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:56,361 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536076045_0001_1_00_000000_0_10070} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536076361, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:47:56,361 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D1B666A065A948A5803476FE1B5CA44B
2016-05-18 01:47:56,362 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:56,362 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:56,362 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D1B666A065A948A5803476FE1B5CA44B, wasInterrupted=false
2016-05-18 01:47:56,363 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:56,364 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:56,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001_wd/localmode-local-dir/output/attempt_1463536076045_0001_1_00_000000_0_10070/file.out, len=732, isLocal=true
2016-05-18 01:47:56,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:56,365 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:56,365 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:56,365 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:56,365 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D1B666A065A948A5803476FE1B5CA44B
2016-05-18 01:47:56,366 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D1B666A065A948A5803476FE1B5CA44B: Shuffle Runner thread complete
2016-05-18 01:47:56,366 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 44BC9F3A3D1042A0AF558D757E68F816
2016-05-18 01:47:56,366 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false"], id: 2C51194EB45B4C67AFD81BE1A7F7A957
2016-05-18 01:47:56,367 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false"]
2016-05-18 01:47:56,367 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 44BC9F3A3D1042A0AF558D757E68F816, for 1 inputs
2016-05-18 01:47:56,367 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C3426BED15574EFAAD55210CE2C8BF0B, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:56,368 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D1B666A065A948A5803476FE1B5CA44B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:56,374 INFO output.MROutput (MROutput.java:close(526)) - 2C51194EB45B4C67AFD81BE1A7F7A957 closed
2016-05-18 01:47:56,374 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536076045_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:56,375 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,375 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C3426BED15574EFAAD55210CE2C8BF0B, mem on close (mb), free: 173, total: 356, max: 672
2016-05-18 01:47:56,375 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D1B666A065A948A5803476FE1B5CA44B
2016-05-18 01:47:56,376 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D1B666A065A948A5803476FE1B5CA44B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:56,376 INFO output.MROutput (MROutput.java:close(526)) - 2C51194EB45B4C67AFD81BE1A7F7A957 closed
2016-05-18 01:47:56,376 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:56 Completed running task attempt: attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,377 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:56,377 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536076045_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:56,378 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536076045_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536076368, Process_Duration=7, Process_End_Time=1463536076375, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:56,379 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536076045_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,379 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, taskAttemptId=attempt_1463536076045_0001_1_01_000000_0, creationTime=1463536076310, allocationTime=1463536076311, startTime=1463536076315, finishTime=1463536076379, timeTaken=64, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:56,380 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536076045_0001_1_01_000000_0
2016-05-18 01:47:56,379 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,380 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:56,381 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,380 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:TASK_FINISHED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, taskId=task_1463536076045_0001_1_01_000000, startTime=1463536076315, finishTime=1463536076380, timeTaken=65, status=SUCCEEDED, successfulAttemptID=attempt_1463536076045_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536076368, Process_Duration=7, Process_End_Time=1463536076375, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:56,382 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:56,382 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B]
2016-05-18 01:47:56,383 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:VERTEX_FINISHED]: vertexName=C3426BED15574EFAAD55210CE2C8BF0B, vertexId=vertex_1463536076045_0001_1_01, initRequestedTime=1463536076224, initedTime=1463536076230, startRequestedTime=1463536076240, startedTime=1463536076241, finishTime=1463536076382, timeTaken=141, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536076368, Process_Duration=7, Process_End_Time=1463536076375, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536076315, firstTasksToStart=[ task_1463536076045_0001_1_01_000000 ], lastTaskFinishTime=1463536076379, lastTasksToFinish=[ task_1463536076045_0001_1_01_000000 ], minTaskDuration=64, maxTaskDuration=64, avgTaskDuration=64.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536076045_0001_1_01_000000 ], longestDurationTasks=[ task_1463536076045_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:56,383 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:56,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536076045_0001_00_000002, stoppedTime=1463536076384, exitStatus=0
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536076045_0001_1_00 [D1B666A065A948A5803476FE1B5CA44B]
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536076045_0001_1
2016-05-18 01:47:56,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536076045_0001_1
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 2C51194EB45B4C67AFD81BE1A7F7A957 for vertex: vertex_1463536076045_0001_1_01 [C3426BED15574EFAAD55210CE2C8BF0B], outputName: 2C51194EB45B4C67AFD81BE1A7F7A957
2016-05-18 01:47:56,386 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076045_0001_00_000002
2016-05-18 01:47:56,387 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536076045_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:56,387 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076045_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:56,387 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536076045_0001_00_000002, Exiting
2016-05-18 01:47:56,388 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536076045_0001_00_000002
2016-05-18 01:47:56,388 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536076045_0001_00_000002
2016-05-18 01:47:56,384 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076045_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:56,388 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536076045_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:56,388 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536076045_0001_00_000002 completed successfully
2016-05-18 01:47:56,389 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463536076045_0001_00_000002
2016-05-18 01:47:56,398 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:2C51194EB45B4C67AFD81BE1A7F7A957 of vertex/vertexGroup:C3426BED15574EFAAD55210CE2C8BF0B isVertexGroupOutput:false
2016-05-18 01:47:56,398 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:56,400 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536076045_0001_1, queueSize=0
2016-05-18 01:47:56,401 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076045_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536076045_0001_1, startTime=1463536076222, finishTime=1463536076398, timeTaken=176, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072152657, Process_Duration=12, Process_End_Time=2927072152669, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:56,401 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536076045_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:56,401 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076045_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:56 Completed Dag: dag_1463536076045_0001_1
2016-05-18 01:47:56,402 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536076045_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:56,402 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536076045_0001_1_post
2016-05-18 01:47:56,402 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,402 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:56,402 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:56,402 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false, with id=dag_1463536076045_0001_1
2016-05-18 01:47:56,402 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420] comprehensivesort/(1/1) ...ve/false_false_true_false, with id=dag_1463536076045_0001_1
2016-05-18 01:47:56,412 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ve/false_false_true_false, applicationId=application_1463536076045_0001
2016-05-18 01:47:56,415 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:56,415 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:56,415 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:56,415 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:56,415 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:56,418 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:56,419 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:56,420 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:56,420 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:56,420 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:56,421 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:56,421 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:56,422 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:56,423 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:56,427 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8CBA9743DC7A4135B19991FE02F3DCCF/F363CFAEEFE44675934C3757C0EA4420/.tez/application_1463536076045_0001
2016-05-18 01:47:56,427 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false/_temporary
2016-05-18 01:47:56,428 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.412
2016-05-18 01:47:56,531 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:56,532 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:56,573 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 01:47:56,585 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.027
2016-05-18 01:47:56,586 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:56,586 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:56,586 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:56,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:56,608 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false
2016-05-18 01:47:56,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:56,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false"]
2016-05-18 01:47:56,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:56,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:56,665 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:56,677 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/true_false_true_false
2016-05-18 01:47:56,677 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:56,692 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:56,697 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:56,697 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:56,697 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:56,706 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001 doesn't exist and is created
2016-05-18 01:47:56,726 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:56,726 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:56,727 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd
2016-05-18 01:47:56,743 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536076697_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:56,743 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:56,744 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:56,745 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:56,745 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:56,745 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:56,745 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:56,746 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:56,747 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:56,747 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:56,747 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:56,747 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:56,748 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:56,748 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:56,748 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:56,748 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:56,748 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:56,748 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:56,749 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:56,749 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:56,750 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:56,751 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:56,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:56,751 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-log-dir
2016-05-18 01:47:56,752 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-log-dir/history.txt.appattempt_1463536076697_0001_000000, maxErrors=10
2016-05-18 01:47:56,752 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:56,752 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536076697_0001_000000, appSubmitTime=1463536076743, launchTime=1463536076743
2016-05-18 01:47:56,756 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:56,760 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:40690
2016-05-18 01:47:56,760 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536076697
2016-05-18 01:47:56,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536076697_0001_000000, startTime=1463536076762
2016-05-18 01:47:56,762 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:56,827 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:56,827 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:56,827 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/true_false_true_false, applicationId=application_1463536076697_0001, dagName=[9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false
2016-05-18 01:47:56,845 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false
2016-05-18 01:47:56,845 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:56,846 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536076697_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-log-dir/dag_1463536076697_0001_1.dot
2016-05-18 01:47:56,847 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-log-dir/dag_1463536076697_0001_1-tez-dag.pb.txt
2016-05-18 01:47:56,853 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536076697_0001_1
2016-05-18 01:47:56,854 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,854 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false
2016-05-18 01:47:56 Running Dag: dag_1463536076697_0001_1
2016-05-18 01:47:56,868 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536076697_0001_1, submitTime=1463536076845
2016-05-18 01:47:56,870 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001/tez-conf.pb" } size: 40937 timestamp: 1463536076000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:56,870 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,870 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:56,871 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001/tez-conf.pb" } size: 40937 timestamp: 1463536076000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:56,871 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,871 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:56,871 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:56,872 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536076697_0001_1, initTime=1463536076872
2016-05-18 01:47:56,872 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076697_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:56,872 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:56,873 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/true_false_true_false, applicationId=application_1463536076697_0001, dagName=[9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false
2016-05-18 01:47:56,873 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:DAG_STARTED]: dagID=dag_1463536076697_0001_1, startTime=1463536076873
2016-05-18 01:47:56,873 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076697_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:56,873 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B073C81056634806B96F9B023A6C2DC5 : {3E634BA9FE8E4E5181027141BF630A5D={InputName=3E634BA9FE8E4E5181027141BF630A5D}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:56,873 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 3E634BA9FE8E4E5181027141BF630A5D, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:56,873 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,874 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,874 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,874 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,874 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:56,875 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,876 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 3E634BA9FE8E4E5181027141BF630A5D on vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,878 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:56,878 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,879 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,879 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 3E634BA9FE8E4E5181027141BF630A5D asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:56,879 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536076697_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:56,880 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3], numAdditionalOutputs=1
2016-05-18 01:47:56,880 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6757A25D3B084FD3940A1BD103D97921, vertex=vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:56,882 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:56,882 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:56,883 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:56,883 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:56,884 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 3E634BA9FE8E4E5181027141BF630A5D on vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,885 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FC57069637E640A086A9AF2495EE9DD3:6757A25D3B084FD3940A1BD103D97921 using oldmapred API
2016-05-18 01:47:56,885 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6757A25D3B084FD3940A1BD103D97921, vertexName=FC57069637E640A086A9AF2495EE9DD3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:56,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, vertexId=vertex_1463536076697_0001_1_01, initRequestedTime=1463536076875, initedTime=1463536076880, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:56,887 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:56,889 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:56,889 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {3E634BA9FE8E4E5181027141BF630A5D=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:56,889 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] parallelism set to 1
2016-05-18 01:47:56,890 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:56,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B073C81056634806B96F9B023A6C2DC5, vertexId=vertex_1463536076697_0001_1_00, initRequestedTime=1463536076873, initedTime=1463536076890, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:56,890 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] with distanceFromRoot: 0
2016-05-18 01:47:56,890 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:56,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536076697_0001_1_00, startRequestedTime=1463536076875, startedTime=1463536076891
2016-05-18 01:47:56,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536076697_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:56,891 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:56,891 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536076697_0001_1_00 for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] numStartedSources: 1 numSources: 1
2016-05-18 01:47:56,891 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] with distanceFromRoot: 1
2016-05-18 01:47:56,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536076697_0001_1_01, startRequestedTime=1463536076891, startedTime=1463536076891
2016-05-18 01:47:56,892 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:56,895 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B073C81056634806B96F9B023A6C2DC5
2016-05-18 01:47:56,896 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FC57069637E640A086A9AF2495EE9DD3 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:56,896 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,896 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B073C81056634806B96F9B023A6C2DC5 in vertex: FC57069637E640A086A9AF2495EE9DD3 numBipartiteSourceTasks: 1
2016-05-18 01:47:56,897 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536076697_0001, with dag id: dag_1463536076697_0001_1
2016-05-18 01:47:56,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_STARTED]: vertexName=B073C81056634806B96F9B023A6C2DC5, taskId=task_1463536076697_0001_1_00_000000, scheduledTime=1463536076897, launchTime=1463536076897
2016-05-18 01:47:56,898 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,899 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:56,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536076697_0001_00_000001, launchTime=1463536076900
2016-05-18 01:47:56,902 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076697_0001_00_000001
2016-05-18 01:47:56,905 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076697_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,905 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536076697_0001_1_00_000000_0] started. Is using containerId: [container_1463536076697_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:56,906 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B073C81056634806B96F9B023A6C2DC5, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0, startTime=1463536076905, containerId=container_1463536076697_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:56 Starting to run new task attempt: attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,906 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,906 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,906 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:56,906 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false, VertexName: B073C81056634806B96F9B023A6C2DC5, VertexParallelism: 1, TaskAttemptID:attempt_1463536076697_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3E634BA9FE8E4E5181027141BF630A5D, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FC57069637E640A086A9AF2495EE9DD3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:56,907 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:56,907 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,907 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536076697_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:56,908 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536076697_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:56,911 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:56,912 INFO input.MRInput (MRInput.java:initialize(440)) - 3E634BA9FE8E4E5181027141BF630A5D using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:56,912 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 3E634BA9FE8E4E5181027141BF630A5D
2016-05-18 01:47:56,917 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B073C81056634806B96F9B023A6C2DC5, ordinal: 0
2016-05-18 01:47:56,917 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B073C81056634806B96F9B023A6C2DC5, mem on start (mb), free: 276, total: 356, max: 672
2016-05-18 01:47:56,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:56,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:56,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:56,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:56,918 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[FC57069637E640A086A9AF2495EE9DD3:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [3E634BA9FE8E4E5181027141BF630A5D:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 01:47:56,918 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:56,918 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:56,918 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:56,919 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[FC57069637E640A086A9AF2495EE9DD3:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [3E634BA9FE8E4E5181027141BF630A5D:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 01:47:56,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:56,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 3E634BA9FE8E4E5181027141BF630A5D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:56,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:56,920 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:56,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 3E634BA9FE8E4E5181027141BF630A5D
2016-05-18 01:47:56,920 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:56,921 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,922 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:56,922 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 3E634BA9FE8E4E5181027141BF630A5D initialized RecordReader from event
2016-05-18 01:47:56,925 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,925 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 3E634BA9FE8E4E5181027141BF630A5D
2016-05-18 01:47:56,925 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 46E4BB4A674041E0A4F2D9D866B79F4C
2016-05-18 01:47:56,926 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 46E4BB4A674041E0A4F2D9D866B79F4C
2016-05-18 01:47:56,928 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FC57069637E640A086A9AF2495EE9DD3 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4b720ff4, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@2d7582de, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:56,929 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FC57069637E640A086A9AF2495EE9DD3: , UsingHashComparator=false
2016-05-18 01:47:56,941 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:56,942 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:56,942 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:56,942 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:56,942 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FC57069637E640A086A9AF2495EE9DD3: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:56,942 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FC57069637E640A086A9AF2495EE9DD3: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:56,943 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:56,943 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B073C81056634806B96F9B023A6C2DC5, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:56,948 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,948 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B073C81056634806B96F9B023A6C2DC5, mem on close (mb), free: 174, total: 356, max: 672
2016-05-18 01:47:56,948 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FC57069637E640A086A9AF2495EE9DD3: Starting flush of map output
2016-05-18 01:47:56,948 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FC57069637E640A086A9AF2495EE9DD3: Span0.length = 37, perItem = 24
2016-05-18 01:47:56,950 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FC57069637E640A086A9AF2495EE9DD3: done sorting span=0, length=37, time=2
2016-05-18 01:47:56,950 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FC57069637E640A086A9AF2495EE9DD3: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:56,951 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FC57069637E640A086A9AF2495EE9DD3: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-local-dir/output/attempt_1463536076697_0001_1_00_000000_0_10077_0/file.out
2016-05-18 01:47:56,955 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:56 Completed running task attempt: attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,956 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:56,956 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536076697_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:56,956 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536076697_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536076943, Process_Duration=5, Process_End_Time=1463536076948, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:56,957 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536076697_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,957 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:56,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B073C81056634806B96F9B023A6C2DC5, taskAttemptId=attempt_1463536076697_0001_1_00_000000_0, creationTime=1463536076898, allocationTime=1463536076899, startTime=1463536076905, finishTime=1463536076957, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:56,958 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536076697_0001_1_00_000000_0
2016-05-18 01:47:56,958 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_FINISHED]: vertexName=B073C81056634806B96F9B023A6C2DC5, taskId=task_1463536076697_0001_1_00_000000, startTime=1463536076905, finishTime=1463536076958, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463536076697_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536076943, Process_Duration=5, Process_End_Time=1463536076948, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:56,959 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:56,959 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:56,958 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:56,960 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,960 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_FINISHED]: vertexName=B073C81056634806B96F9B023A6C2DC5, vertexId=vertex_1463536076697_0001_1_00, initRequestedTime=1463536076873, initedTime=1463536076890, startRequestedTime=1463536076875, startedTime=1463536076891, finishTime=1463536076959, timeTaken=68, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536076943, Process_Duration=5, Process_End_Time=1463536076948, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536076905, firstTasksToStart=[ task_1463536076697_0001_1_00_000000 ], lastTaskFinishTime=1463536076957, lastTasksToFinish=[ task_1463536076697_0001_1_00_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536076697_0001_1_00_000000 ], longestDurationTasks=[ task_1463536076697_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:56,960 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:56,960 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] attempt: attempt_1463536076697_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:56,960 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:56,961 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:56,961 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076697_0001_00_000001
2016-05-18 01:47:56,962 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536076697_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:56,962 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536076697_0001_00_000001, stoppedTime=1463536076962, exitStatus=0
2016-05-18 01:47:56,961 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FC57069637E640A086A9AF2495EE9DD3
2016-05-18 01:47:56,963 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536076697_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:56,962 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076697_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:56,963 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B073C81056634806B96F9B023A6C2DC5, destinationVertexName=FC57069637E640A086A9AF2495EE9DD3. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:56,963 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,963 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536076697_0001_00_000001, Exiting
2016-05-18 01:47:56,964 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536076697_0001_00_000001
2016-05-18 01:47:56,964 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536076697_0001_00_000001
2016-05-18 01:47:56,964 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_STARTED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, taskId=task_1463536076697_0001_1_01_000000, scheduledTime=1463536076964, launchTime=1463536076964
2016-05-18 01:47:56,964 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,965 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536076697_0001_00_000002, launchTime=1463536076965
2016-05-18 01:47:56,966 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536076697_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:56,966 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536076697_0001_00_000001 completed successfully
2016-05-18 01:47:56,966 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076697_0001_00_000002
2016-05-18 01:47:56,967 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076697_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:56,967 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536076697_0001_1_01_000000_0] started. Is using containerId: [container_1463536076697_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:56,967 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0, startTime=1463536076967, containerId=container_1463536076697_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:56 Starting to run new task attempt: attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:56,968 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:56,968 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:56,968 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:56,969 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false, VertexName: FC57069637E640A086A9AF2495EE9DD3, VertexParallelism: 1, TaskAttemptID:attempt_1463536076697_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B073C81056634806B96F9B023A6C2DC5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6757A25D3B084FD3940A1BD103D97921, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:56,969 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:56,969 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:56,970 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536076697_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:56,970 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536076697_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:56,972 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:56,975 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:56,983 INFO output.MROutput (MROutput.java:initialize(421)) - 6757A25D3B084FD3940A1BD103D97921: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:56,984 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FC57069637E640A086A9AF2495EE9DD3, ordinal: 1
2016-05-18 01:47:56,984 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC57069637E640A086A9AF2495EE9DD3, mem on start (mb), free: 168, total: 356, max: 672
2016-05-18 01:47:56,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:56,985 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:56,985 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:56,985 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:56,985 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6757A25D3B084FD3940A1BD103D97921:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B073C81056634806B96F9B023A6C2DC5:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:56,986 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:56,986 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:56,986 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:56,987 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6757A25D3B084FD3940A1BD103D97921:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B073C81056634806B96F9B023A6C2DC5:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:56,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:56,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B073C81056634806B96F9B023A6C2DC5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:56,988 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:56,988 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:56,988 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B073C81056634806B96F9B023A6C2DC5: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:56,989 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B073C81056634806B96F9B023A6C2DC5: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:56,990 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B073C81056634806B96F9B023A6C2DC5 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:56,992 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B073C81056634806B96F9B023A6C2DC5
2016-05-18 01:47:56,992 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:56,993 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:56,994 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B073C81056634806B96F9B023A6C2DC5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:56,996 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536076697_0001_1_00_000000_0_10077} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536076996, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:47:56,996 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B073C81056634806B96F9B023A6C2DC5
2016-05-18 01:47:56,997 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:56,997 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:56,997 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B073C81056634806B96F9B023A6C2DC5, wasInterrupted=false
2016-05-18 01:47:56,998 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 46E4BB4A674041E0A4F2D9D866B79F4C
2016-05-18 01:47:56,999 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false"], id: 6757A25D3B084FD3940A1BD103D97921
2016-05-18 01:47:56,999 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false"]
2016-05-18 01:47:56,999 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 46E4BB4A674041E0A4F2D9D866B79F4C, for 1 inputs
2016-05-18 01:47:57,001 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:57,001 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:57,002 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001_wd/localmode-local-dir/output/attempt_1463536076697_0001_1_00_000000_0_10077/file.out, len=732, isLocal=true
2016-05-18 01:47:57,002 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:57,002 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:57,002 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:57,003 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:57,003 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B073C81056634806B96F9B023A6C2DC5
2016-05-18 01:47:57,003 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B073C81056634806B96F9B023A6C2DC5: Shuffle Runner thread complete
2016-05-18 01:47:57,003 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FC57069637E640A086A9AF2495EE9DD3, all 1 inputs ready in: 00:00:00.003
2016-05-18 01:47:57,004 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B073C81056634806B96F9B023A6C2DC5: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:57,011 INFO output.MROutput (MROutput.java:close(526)) - 6757A25D3B084FD3940A1BD103D97921 closed
2016-05-18 01:47:57,011 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536076697_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:57,012 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:57,012 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC57069637E640A086A9AF2495EE9DD3, mem on close (mb), free: 164, total: 356, max: 672
2016-05-18 01:47:57,013 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B073C81056634806B96F9B023A6C2DC5
2016-05-18 01:47:57,013 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B073C81056634806B96F9B023A6C2DC5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:57,013 INFO output.MROutput (MROutput.java:close(526)) - 6757A25D3B084FD3940A1BD103D97921 closed
2016-05-18 01:47:57,013 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:57 Completed running task attempt: attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:57,014 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:57,015 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536076697_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:57,015 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536076697_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536077003, Process_Duration=9, Process_End_Time=1463536077012, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:57,016 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536076697_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, taskAttemptId=attempt_1463536076697_0001_1_01_000000_0, creationTime=1463536076964, allocationTime=1463536076965, startTime=1463536076967, finishTime=1463536077017, timeTaken=50, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:57,017 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,017 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:57,017 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536076697_0001_1_01_000000_0
2016-05-18 01:47:57,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:TASK_FINISHED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, taskId=task_1463536076697_0001_1_01_000000, startTime=1463536076967, finishTime=1463536077017, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463536076697_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536077003, Process_Duration=9, Process_End_Time=1463536077012, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:57,018 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:57,018 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3]
2016-05-18 01:47:57,018 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,019 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536076697_0001_00_000002
2016-05-18 01:47:57,019 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:VERTEX_FINISHED]: vertexName=FC57069637E640A086A9AF2495EE9DD3, vertexId=vertex_1463536076697_0001_1_01, initRequestedTime=1463536076875, initedTime=1463536076880, startRequestedTime=1463536076891, startedTime=1463536076891, finishTime=1463536077018, timeTaken=127, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536077003, Process_Duration=9, Process_End_Time=1463536077012, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536076967, firstTasksToStart=[ task_1463536076697_0001_1_01_000000 ], lastTaskFinishTime=1463536077017, lastTasksToFinish=[ task_1463536076697_0001_1_01_000000 ], minTaskDuration=50, maxTaskDuration=50, avgTaskDuration=50.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536076697_0001_1_01_000000 ], longestDurationTasks=[ task_1463536076697_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:57,020 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:57,020 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536076697_0001_00_000002, stoppedTime=1463536077020, exitStatus=0
2016-05-18 01:47:57,020 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:57,020 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:57,020 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536076697_0001_1_00 [B073C81056634806B96F9B023A6C2DC5]
2016-05-18 01:47:57,020 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536076697_0001_1
2016-05-18 01:47:57,021 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536076697_0001_1
2016-05-18 01:47:57,021 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076697_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:57,021 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6757A25D3B084FD3940A1BD103D97921 for vertex: vertex_1463536076697_0001_1_01 [FC57069637E640A086A9AF2495EE9DD3], outputName: 6757A25D3B084FD3940A1BD103D97921
2016-05-18 01:47:57,021 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536076697_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:57,030 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536076697_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:57,030 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536076697_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:57,030 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536076697_0001_00_000002, Exiting
2016-05-18 01:47:57,030 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536076697_0001_00_000002
2016-05-18 01:47:57,030 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536076697_0001_00_000002
2016-05-18 01:47:57,030 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536076697_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:57,030 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536076697_0001_00_000002 completed successfully
2016-05-18 01:47:57,032 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6757A25D3B084FD3940A1BD103D97921 of vertex/vertexGroup:FC57069637E640A086A9AF2495EE9DD3 isVertexGroupOutput:false
2016-05-18 01:47:57,032 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:57,035 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536076697_0001_1, queueSize=0
2016-05-18 01:47:57,036 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536076697_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536076697_0001_1, startTime=1463536076872, finishTime=1463536077033, timeTaken=161, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=310, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072153946, Process_Duration=14, Process_End_Time=2927072153960, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:57,036 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536076697_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:57,036 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536076697_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:57 Completed Dag: dag_1463536076697_0001_1
2016-05-18 01:47:57,038 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536076697_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:57,038 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536076697_0001_1_post
2016-05-18 01:47:57,039 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,040 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:57,040 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:57,041 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false, with id=dag_1463536076697_0001_1
2016-05-18 01:47:57,041 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1] comprehensivesort/(1/1) ...ive/true_false_true_false, with id=dag_1463536076697_0001_1
2016-05-18 01:47:57,043 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/true_false_true_false, applicationId=application_1463536076697_0001
2016-05-18 01:47:57,045 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:57,045 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:57,045 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:57,046 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:57,046 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:57,046 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:57,047 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:57,047 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:57,047 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:57,047 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:57,047 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:57,048 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:57,048 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:57,048 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:57,051 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/9305C7A2C1A0490987C7E378B236342D/D73221FE9EC24D0E8BFC1B31D96A0AB1/.tez/application_1463536076697_0001
2016-05-18 01:47:57,052 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false/_temporary
2016-05-18 01:47:57,053 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.388
2016-05-18 01:47:57,126 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:57,127 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:57,170 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 01:47:57,183 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.028
2016-05-18 01:47:57,184 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:57,184 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:57,184 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:57,200 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:57,211 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false
2016-05-18 01:47:57,285 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false"]
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:57,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/true_true_true_false
2016-05-18 01:47:57,287 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:57,305 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:57,309 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:57,310 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:57,310 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:57,319 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001 doesn't exist and is created
2016-05-18 01:47:57,341 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:57,341 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:57,342 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd
2016-05-18 01:47:57,360 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536077310_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:57,360 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:57,360 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:57,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:57,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:57,362 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:57,362 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:57,363 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:57,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:57,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:57,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:57,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:57,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:57,366 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:57,366 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:57,366 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:57,366 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:57,367 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:57,367 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:57,367 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:57,369 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:57,370 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:57,370 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:57,370 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-log-dir
2016-05-18 01:47:57,371 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-log-dir/history.txt.appattempt_1463536077310_0001_000000, maxErrors=10
2016-05-18 01:47:57,371 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:57,371 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536077310_0001_000000, appSubmitTime=1463536077359, launchTime=1463536077359
2016-05-18 01:47:57,377 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42603
2016-05-18 01:47:57,378 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536077310
2016-05-18 01:47:57,380 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:57,383 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536077310_0001_000000, startTime=1463536077383
2016-05-18 01:47:57,383 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:57,441 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:57,442 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:57,442 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/true_true_true_false, applicationId=application_1463536077310_0001, dagName=[0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false
2016-05-18 01:47:57,463 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false
2016-05-18 01:47:57,463 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:57,464 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536077310_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-log-dir/dag_1463536077310_0001_1.dot
2016-05-18 01:47:57,465 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-log-dir/dag_1463536077310_0001_1-tez-dag.pb.txt
2016-05-18 01:47:57,471 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536077310_0001_1
2016-05-18 01:47:57,472 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,472 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false
2016-05-18 01:47:57 Running Dag: dag_1463536077310_0001_1
2016-05-18 01:47:57,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536077310_0001_1, submitTime=1463536077463
2016-05-18 01:47:57,488 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001/tez-conf.pb" } size: 40937 timestamp: 1463536077000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:57,489 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,489 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:57,490 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001/tez-conf.pb" } size: 40937 timestamp: 1463536077000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:57,490 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,491 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:57,491 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:57,492 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536077310_0001_1, initTime=1463536077491
2016-05-18 01:47:57,492 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077310_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:57,492 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:57,493 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:DAG_STARTED]: dagID=dag_1463536077310_0001_1, startTime=1463536077493
2016-05-18 01:47:57,493 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077310_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:57,493 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DB682537D09F4F00B55A1C4BC06BF3AE : {8547AA8F87DB41BEA05FD3B03E232BB2={InputName=8547AA8F87DB41BEA05FD3B03E232BB2}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:57,493 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 8547AA8F87DB41BEA05FD3B03E232BB2, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:57,493 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,493 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,494 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,494 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,494 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/true_true_true_false, applicationId=application_1463536077310_0001, dagName=[0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false
2016-05-18 01:47:57,495 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:57,495 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 8547AA8F87DB41BEA05FD3B03E232BB2 on vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,495 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,497 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 8547AA8F87DB41BEA05FD3B03E232BB2 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:57,499 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:57,499 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,499 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:57,499 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:57,499 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536077310_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:57,500 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6], numAdditionalOutputs=1
2016-05-18 01:47:57,501 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=A56524E478DB4A5F8B584562901F8A50, vertex=vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:57,500 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:57,501 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:57,502 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 8547AA8F87DB41BEA05FD3B03E232BB2 on vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,503 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B758575C2224461B918DF81274F4AAC6:A56524E478DB4A5F8B584562901F8A50 using oldmapred API
2016-05-18 01:47:57,504 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=A56524E478DB4A5F8B584562901F8A50, vertexName=B758575C2224461B918DF81274F4AAC6, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:57,505 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B758575C2224461B918DF81274F4AAC6, vertexId=vertex_1463536077310_0001_1_01, initRequestedTime=1463536077495, initedTime=1463536077500, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:57,505 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:57,506 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:57,506 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {8547AA8F87DB41BEA05FD3B03E232BB2=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:57,507 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] parallelism set to 1
2016-05-18 01:47:57,507 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:57,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, vertexId=vertex_1463536077310_0001_1_00, initRequestedTime=1463536077493, initedTime=1463536077507, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:57,508 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] with distanceFromRoot: 0
2016-05-18 01:47:57,508 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:57,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536077310_0001_1_00, startRequestedTime=1463536077495, startedTime=1463536077508
2016-05-18 01:47:57,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536077310_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:57,509 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:57,509 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536077310_0001_1_00 for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] numStartedSources: 1 numSources: 1
2016-05-18 01:47:57,509 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] with distanceFromRoot: 1
2016-05-18 01:47:57,509 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DB682537D09F4F00B55A1C4BC06BF3AE
2016-05-18 01:47:57,510 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,511 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536077310_0001_1_01, startRequestedTime=1463536077509, startedTime=1463536077510
2016-05-18 01:47:57,511 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:57,513 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_STARTED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, taskId=task_1463536077310_0001_1_00_000000, scheduledTime=1463536077512, launchTime=1463536077512
2016-05-18 01:47:57,513 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,514 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B758575C2224461B918DF81274F4AAC6 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:57,514 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:57,516 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DB682537D09F4F00B55A1C4BC06BF3AE in vertex: B758575C2224461B918DF81274F4AAC6 numBipartiteSourceTasks: 1
2016-05-18 01:47:57,518 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536077310_0001_00_000001, launchTime=1463536077518
2016-05-18 01:47:57,519 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077310_0001_00_000001
2016-05-18 01:47:57,519 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536077310_0001, with dag id: dag_1463536077310_0001_1
2016-05-18 01:47:57,520 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077310_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,520 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536077310_0001_1_00_000000_0] started. Is using containerId: [container_1463536077310_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:57,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0, startTime=1463536077520, containerId=container_1463536077310_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:57 Starting to run new task attempt: attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,522 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,522 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,522 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:57,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false, VertexName: DB682537D09F4F00B55A1C4BC06BF3AE, VertexParallelism: 1, TaskAttemptID:attempt_1463536077310_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=8547AA8F87DB41BEA05FD3B03E232BB2, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B758575C2224461B918DF81274F4AAC6, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:57,523 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:57,526 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,526 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536077310_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:57,527 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536077310_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:57,529 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:57,532 INFO input.MRInput (MRInput.java:initialize(440)) - 8547AA8F87DB41BEA05FD3B03E232BB2 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:57,533 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 8547AA8F87DB41BEA05FD3B03E232BB2
2016-05-18 01:47:57,537 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DB682537D09F4F00B55A1C4BC06BF3AE, ordinal: 0
2016-05-18 01:47:57,537 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB682537D09F4F00B55A1C4BC06BF3AE, mem on start (mb), free: 269, total: 356, max: 672
2016-05-18 01:47:57,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:57,537 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:57,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:57,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:57,538 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[8547AA8F87DB41BEA05FD3B03E232BB2:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B758575C2224461B918DF81274F4AAC6:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:57,538 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:57,538 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:57,538 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:57,538 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[8547AA8F87DB41BEA05FD3B03E232BB2:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B758575C2224461B918DF81274F4AAC6:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:57,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:57,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 8547AA8F87DB41BEA05FD3B03E232BB2 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:57,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:57,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:57,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 8547AA8F87DB41BEA05FD3B03E232BB2
2016-05-18 01:47:57,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:57,540 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,543 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:57,544 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:57,545 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 8547AA8F87DB41BEA05FD3B03E232BB2 initialized RecordReader from event
2016-05-18 01:47:57,544 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 8547AA8F87DB41BEA05FD3B03E232BB2
2016-05-18 01:47:57,545 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 28AF49F607F748DB8465902C5BB2D113
2016-05-18 01:47:57,545 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 28AF49F607F748DB8465902C5BB2D113
2016-05-18 01:47:57,549 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B758575C2224461B918DF81274F4AAC6 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1125d6d3, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@7dee78a5, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:57,549 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B758575C2224461B918DF81274F4AAC6: , UsingHashComparator=false
2016-05-18 01:47:57,562 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:57,562 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:57,562 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B758575C2224461B918DF81274F4AAC6: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:57,562 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B758575C2224461B918DF81274F4AAC6: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:57,563 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:57,563 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DB682537D09F4F00B55A1C4BC06BF3AE, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:57,572 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,572 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB682537D09F4F00B55A1C4BC06BF3AE, mem on close (mb), free: 167, total: 356, max: 672
2016-05-18 01:47:57,572 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B758575C2224461B918DF81274F4AAC6: Starting flush of map output
2016-05-18 01:47:57,573 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B758575C2224461B918DF81274F4AAC6: Span0.length = 37, perItem = 24
2016-05-18 01:47:57,574 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B758575C2224461B918DF81274F4AAC6: done sorting span=0, length=37, time=1
2016-05-18 01:47:57,575 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B758575C2224461B918DF81274F4AAC6: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:57,576 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B758575C2224461B918DF81274F4AAC6: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-local-dir/output/attempt_1463536077310_0001_1_00_000000_0_10083_0/file.out
2016-05-18 01:47:57,579 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:57 Completed running task attempt: attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,579 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:57,580 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536077310_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:57,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536077310_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536077563, Process_Duration=9, Process_End_Time=1463536077572, Read_Duration=6, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:57,581 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536077310_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,581 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,581 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:57,581 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, taskAttemptId=attempt_1463536077310_0001_1_00_000000_0, creationTime=1463536077513, allocationTime=1463536077517, startTime=1463536077520, finishTime=1463536077581, timeTaken=61, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:57,582 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536077310_0001_1_00_000000_0
2016-05-18 01:47:57,582 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,583 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_FINISHED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, taskId=task_1463536077310_0001_1_00_000000, startTime=1463536077520, finishTime=1463536077582, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1463536077310_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536077563, Process_Duration=9, Process_End_Time=1463536077572, Read_Duration=6, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:57,583 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:57,583 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,584 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_FINISHED]: vertexName=DB682537D09F4F00B55A1C4BC06BF3AE, vertexId=vertex_1463536077310_0001_1_00, initRequestedTime=1463536077493, initedTime=1463536077507, startRequestedTime=1463536077495, startedTime=1463536077508, finishTime=1463536077583, timeTaken=75, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536077563, Process_Duration=9, Process_End_Time=1463536077572, Read_Duration=6, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536077520, firstTasksToStart=[ task_1463536077310_0001_1_00_000000 ], lastTaskFinishTime=1463536077581, lastTasksToFinish=[ task_1463536077310_0001_1_00_000000 ], minTaskDuration=61, maxTaskDuration=61, avgTaskDuration=61.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536077310_0001_1_00_000000 ], longestDurationTasks=[ task_1463536077310_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:57,584 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077310_0001_00_000001
2016-05-18 01:47:57,584 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:57,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536077310_0001_00_000001, stoppedTime=1463536077585, exitStatus=0
2016-05-18 01:47:57,585 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] attempt: attempt_1463536077310_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:57,585 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:57,585 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:57,585 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B758575C2224461B918DF81274F4AAC6
2016-05-18 01:47:57,585 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DB682537D09F4F00B55A1C4BC06BF3AE, destinationVertexName=B758575C2224461B918DF81274F4AAC6. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:57,585 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,585 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536077310_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:57,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_STARTED]: vertexName=B758575C2224461B918DF81274F4AAC6, taskId=task_1463536077310_0001_1_01_000000, scheduledTime=1463536077586, launchTime=1463536077586
2016-05-18 01:47:57,586 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,587 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536077310_0001_00_000002, launchTime=1463536077587
2016-05-18 01:47:57,588 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077310_0001_00_000002
2016-05-18 01:47:57,589 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077310_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,589 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536077310_0001_1_01_000000_0] started. Is using containerId: [container_1463536077310_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:57,589 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B758575C2224461B918DF81274F4AAC6, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0, startTime=1463536077588, containerId=container_1463536077310_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:57 Starting to run new task attempt: attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,589 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,590 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,590 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:57,590 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false, VertexName: B758575C2224461B918DF81274F4AAC6, VertexParallelism: 1, TaskAttemptID:attempt_1463536077310_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DB682537D09F4F00B55A1C4BC06BF3AE, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A56524E478DB4A5F8B584562901F8A50, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:57,590 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:57,591 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:57,591 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:57,592 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,593 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536077310_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,593 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536077310_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:57,594 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:57,596 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536077310_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:57,596 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077310_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:57,597 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536077310_0001_00_000001, Exiting
2016-05-18 01:47:57,597 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536077310_0001_00_000001
2016-05-18 01:47:57,597 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536077310_0001_00_000001
2016-05-18 01:47:57,598 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536077310_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:57,598 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536077310_0001_00_000001 completed successfully
2016-05-18 01:47:57,599 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:57,607 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B758575C2224461B918DF81274F4AAC6, ordinal: 1
2016-05-18 01:47:57,607 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B758575C2224461B918DF81274F4AAC6, mem on start (mb), free: 160, total: 356, max: 672
2016-05-18 01:47:57,607 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:57,607 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:57,607 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:57,608 INFO output.MROutput (MROutput.java:initialize(421)) - A56524E478DB4A5F8B584562901F8A50: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:57,608 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:57,608 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A56524E478DB4A5F8B584562901F8A50:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [DB682537D09F4F00B55A1C4BC06BF3AE:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:57,609 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:57,609 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:57,609 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:57,610 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A56524E478DB4A5F8B584562901F8A50:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [DB682537D09F4F00B55A1C4BC06BF3AE:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:57,610 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:57,610 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DB682537D09F4F00B55A1C4BC06BF3AE being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:57,611 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DB682537D09F4F00B55A1C4BC06BF3AE: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:57,611 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DB682537D09F4F00B55A1C4BC06BF3AE: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:57,612 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DB682537D09F4F00B55A1C4BC06BF3AE with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:57,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:57,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DB682537D09F4F00B55A1C4BC06BF3AE
2016-05-18 01:47:57,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:57,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:57,614 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,615 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB682537D09F4F00B55A1C4BC06BF3AE: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:57,617 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536077310_0001_1_00_000000_0_10083} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536077617, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:47:57,619 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DB682537D09F4F00B55A1C4BC06BF3AE
2016-05-18 01:47:57,619 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:57,619 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DB682537D09F4F00B55A1C4BC06BF3AE, wasInterrupted=false
2016-05-18 01:47:57,619 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:57,620 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:57,621 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:57,621 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 28AF49F607F748DB8465902C5BB2D113
2016-05-18 01:47:57,621 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false"], id: A56524E478DB4A5F8B584562901F8A50
2016-05-18 01:47:57,622 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false"]
2016-05-18 01:47:57,622 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 28AF49F607F748DB8465902C5BB2D113, for 1 inputs
2016-05-18 01:47:57,622 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001_wd/localmode-local-dir/output/attempt_1463536077310_0001_1_00_000000_0_10083/file.out, len=732, isLocal=true
2016-05-18 01:47:57,622 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:57,622 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:57,623 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:57,623 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:57,623 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DB682537D09F4F00B55A1C4BC06BF3AE
2016-05-18 01:47:57,623 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DB682537D09F4F00B55A1C4BC06BF3AE: Shuffle Runner thread complete
2016-05-18 01:47:57,623 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B758575C2224461B918DF81274F4AAC6, all 1 inputs ready in: 00:00:00.001
2016-05-18 01:47:57,625 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DB682537D09F4F00B55A1C4BC06BF3AE: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:57,631 INFO output.MROutput (MROutput.java:close(526)) - A56524E478DB4A5F8B584562901F8A50 closed
2016-05-18 01:47:57,631 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536077310_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:57,632 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,632 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B758575C2224461B918DF81274F4AAC6, mem on close (mb), free: 155, total: 356, max: 672
2016-05-18 01:47:57,632 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DB682537D09F4F00B55A1C4BC06BF3AE
2016-05-18 01:47:57,632 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB682537D09F4F00B55A1C4BC06BF3AE: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:57,632 INFO output.MROutput (MROutput.java:close(526)) - A56524E478DB4A5F8B584562901F8A50 closed
2016-05-18 01:47:57,632 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:57 Completed running task attempt: attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,633 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:57,633 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536077310_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:57,634 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536077310_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536077623, Process_Duration=9, Process_End_Time=1463536077632, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:57,635 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536077310_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,635 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:57,635 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B758575C2224461B918DF81274F4AAC6, taskAttemptId=attempt_1463536077310_0001_1_01_000000_0, creationTime=1463536077586, allocationTime=1463536077587, startTime=1463536077588, finishTime=1463536077635, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:57,635 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:57,635 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,635 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536077310_0001_1_01_000000_0
2016-05-18 01:47:57,635 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077310_0001_00_000002
2016-05-18 01:47:57,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:TASK_FINISHED]: vertexName=B758575C2224461B918DF81274F4AAC6, taskId=task_1463536077310_0001_1_01_000000, startTime=1463536077588, finishTime=1463536077635, timeTaken=47, status=SUCCEEDED, successfulAttemptID=attempt_1463536077310_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536077623, Process_Duration=9, Process_End_Time=1463536077632, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:57,636 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:57,636 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6]
2016-05-18 01:47:57,637 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:VERTEX_FINISHED]: vertexName=B758575C2224461B918DF81274F4AAC6, vertexId=vertex_1463536077310_0001_1_01, initRequestedTime=1463536077495, initedTime=1463536077500, startRequestedTime=1463536077509, startedTime=1463536077510, finishTime=1463536077636, timeTaken=126, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536077623, Process_Duration=9, Process_End_Time=1463536077632, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536077588, firstTasksToStart=[ task_1463536077310_0001_1_01_000000 ], lastTaskFinishTime=1463536077635, lastTasksToFinish=[ task_1463536077310_0001_1_01_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536077310_0001_1_01_000000 ], longestDurationTasks=[ task_1463536077310_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:57,637 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:57,638 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536077310_0001_00_000002, stoppedTime=1463536077638, exitStatus=0
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536077310_0001_1_00 [DB682537D09F4F00B55A1C4BC06BF3AE]
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536077310_0001_1
2016-05-18 01:47:57,638 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536077310_0001_1
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077310_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:57,638 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: A56524E478DB4A5F8B584562901F8A50 for vertex: vertex_1463536077310_0001_1_01 [B758575C2224461B918DF81274F4AAC6], outputName: A56524E478DB4A5F8B584562901F8A50
2016-05-18 01:47:57,639 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536077310_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:57,646 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536077310_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:57,646 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077310_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:57,646 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536077310_0001_00_000002, Exiting
2016-05-18 01:47:57,646 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536077310_0001_00_000002
2016-05-18 01:47:57,646 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536077310_0001_00_000002
2016-05-18 01:47:57,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536077310_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:57,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536077310_0001_00_000002 completed successfully
2016-05-18 01:47:57,649 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:A56524E478DB4A5F8B584562901F8A50 of vertex/vertexGroup:B758575C2224461B918DF81274F4AAC6 isVertexGroupOutput:false
2016-05-18 01:47:57,649 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:57,651 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536077310_0001_1, queueSize=0
2016-05-18 01:47:57,652 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077310_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536077310_0001_1, startTime=1463536077492, finishTime=1463536077649, timeTaken=157, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=290, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072155186, Process_Duration=18, Process_End_Time=2927072155204, Read_Duration=9, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:57,652 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536077310_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:57,652 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077310_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:57 Completed Dag: dag_1463536077310_0001_1
2016-05-18 01:47:57,652 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536077310_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:57,652 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536077310_0001_1_post
2016-05-18 01:47:57,652 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:57,653 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:57,653 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:57,653 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false, with id=dag_1463536077310_0001_1
2016-05-18 01:47:57,653 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3] comprehensivesort/(1/1) ...sive/true_true_true_false, with id=dag_1463536077310_0001_1
2016-05-18 01:47:57,659 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/true_true_true_false, applicationId=application_1463536077310_0001
2016-05-18 01:47:57,661 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:57,661 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:57,661 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:57,661 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:57,661 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:57,661 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:57,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:57,662 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:57,663 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:57,663 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:57,663 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:57,663 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:57,664 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:57,664 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:57,667 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/0E86B4825A834F11BF2D6FADB489F8BE/A04E24D40D7B4A149BD169A8F60132E3/.tez/application_1463536077310_0001
2016-05-18 01:47:57,667 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false/_temporary
2016-05-18 01:47:57,668 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.382
2016-05-18 01:47:57,746 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:57,746 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:57,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 01:47:57,811 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.027
2016-05-18 01:47:57,812 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:57,812 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:57,812 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:57,828 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:57,835 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false
2016-05-18 01:47:57,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:57,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:57,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false"]
2016-05-18 01:47:57,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:57,905 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:57,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:57,908 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/false_true_true_false
2016-05-18 01:47:57,908 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:57,928 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:57,934 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:57,934 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:57,934 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:57,944 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001 doesn't exist and is created
2016-05-18 01:47:57,965 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:57,965 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:57,966 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd
2016-05-18 01:47:57,982 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536077934_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:57,983 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:57,983 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:57,984 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:57,985 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:57,985 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:57,985 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:57,986 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:57,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:57,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:57,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:57,988 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:57,988 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:57,988 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:57,988 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:57,989 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:57,989 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:57,989 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:57,989 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:57,990 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:57,991 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:57,992 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:57,992 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:57,992 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-log-dir
2016-05-18 01:47:57,993 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-log-dir/history.txt.appattempt_1463536077934_0001_000000, maxErrors=10
2016-05-18 01:47:57,993 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:57,993 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536077934_0001_000000, appSubmitTime=1463536077982, launchTime=1463536077982
2016-05-18 01:47:57,995 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:57,999 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:34991
2016-05-18 01:47:58,000 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536077934
2016-05-18 01:47:58,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536077934_0001_000000, startTime=1463536078002
2016-05-18 01:47:58,003 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:58,065 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:58,065 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:58,066 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/false_true_true_false, applicationId=application_1463536077934_0001, dagName=[8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false
2016-05-18 01:47:58,084 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false
2016-05-18 01:47:58,084 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:58,085 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536077934_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-log-dir/dag_1463536077934_0001_1.dot
2016-05-18 01:47:58,085 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-log-dir/dag_1463536077934_0001_1-tez-dag.pb.txt
2016-05-18 01:47:58,095 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536077934_0001_1
2016-05-18 01:47:58,096 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,096 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false
2016-05-18 01:47:58 Running Dag: dag_1463536077934_0001_1
2016-05-18 01:47:58,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536077934_0001_1, submitTime=1463536078084
2016-05-18 01:47:58,111 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001/tez-conf.pb" } size: 40937 timestamp: 1463536077000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:58,111 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,112 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:58,112 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001/tez-conf.pb" } size: 40937 timestamp: 1463536077000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:58,112 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,113 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:58,113 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:58,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536077934_0001_1, initTime=1463536078113
2016-05-18 01:47:58,114 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077934_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:58,114 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:58,115 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/false_true_true_false, applicationId=application_1463536077934_0001, dagName=[8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false
2016-05-18 01:47:58,115 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:DAG_STARTED]: dagID=dag_1463536077934_0001_1, startTime=1463536078115
2016-05-18 01:47:58,115 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077934_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:58,115 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AEB062708FEB4D7E935B1187B1C9AA1F : {706EE65833C0453491672AB431EE99BD={InputName=706EE65833C0453491672AB431EE99BD}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:58,115 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 706EE65833C0453491672AB431EE99BD, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:58,115 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,116 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,116 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,116 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,117 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:58,117 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 706EE65833C0453491672AB431EE99BD on vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,117 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,120 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 706EE65833C0453491672AB431EE99BD asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:58,121 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:58,122 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,122 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,122 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:58,122 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536077934_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:58,123 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05], numAdditionalOutputs=1
2016-05-18 01:47:58,123 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=8C16FF468DF742CC81461C81E1B223B7, vertex=vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:58,123 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:58,124 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:58,124 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:58,124 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 706EE65833C0453491672AB431EE99BD on vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,129 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E51CE4596DE1464EA1581641C7F92B05:8C16FF468DF742CC81461C81E1B223B7 using oldmapred API
2016-05-18 01:47:58,130 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=8C16FF468DF742CC81461C81E1B223B7, vertexName=E51CE4596DE1464EA1581641C7F92B05, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:58,131 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, vertexId=vertex_1463536077934_0001_1_01, initRequestedTime=1463536078117, initedTime=1463536078123, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:58,131 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:58,132 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:58,132 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {706EE65833C0453491672AB431EE99BD=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:58,132 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] parallelism set to 1
2016-05-18 01:47:58,133 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:58,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, vertexId=vertex_1463536077934_0001_1_00, initRequestedTime=1463536078115, initedTime=1463536078133, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:58,134 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] with distanceFromRoot: 0
2016-05-18 01:47:58,134 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:58,134 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536077934_0001_1_00, startRequestedTime=1463536078117, startedTime=1463536078134
2016-05-18 01:47:58,134 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536077934_0001, with dag id: dag_1463536077934_0001_1
2016-05-18 01:47:58,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536077934_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:58,135 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:58,135 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536077934_0001_1_00 for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] numStartedSources: 1 numSources: 1
2016-05-18 01:47:58,135 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] with distanceFromRoot: 1
2016-05-18 01:47:58,135 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AEB062708FEB4D7E935B1187B1C9AA1F
2016-05-18 01:47:58,136 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536077934_0001_1_01, startRequestedTime=1463536078135, startedTime=1463536078135
2016-05-18 01:47:58,136 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:58,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_STARTED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, taskId=task_1463536077934_0001_1_00_000000, scheduledTime=1463536078139, launchTime=1463536078139
2016-05-18 01:47:58,140 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,140 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E51CE4596DE1464EA1581641C7F92B05 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:58,142 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AEB062708FEB4D7E935B1187B1C9AA1F in vertex: E51CE4596DE1464EA1581641C7F92B05 numBipartiteSourceTasks: 1
2016-05-18 01:47:58,143 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:58,145 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536077934_0001_00_000001, launchTime=1463536078145
2016-05-18 01:47:58,145 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077934_0001_00_000001
2016-05-18 01:47:58,146 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536077934_0001_1_00_000000_0] started. Is using containerId: [container_1463536077934_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:58,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0, startTime=1463536078146, containerId=container_1463536077934_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:58,147 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077934_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58 Starting to run new task attempt: attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,147 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,147 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,148 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:58,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false, VertexName: AEB062708FEB4D7E935B1187B1C9AA1F, VertexParallelism: 1, TaskAttemptID:attempt_1463536077934_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=706EE65833C0453491672AB431EE99BD, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E51CE4596DE1464EA1581641C7F92B05, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:58,148 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:58,148 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,152 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536077934_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:58,152 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536077934_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:58,153 INFO input.MRInput (MRInput.java:initialize(440)) - 706EE65833C0453491672AB431EE99BD using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:58,154 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 706EE65833C0453491672AB431EE99BD
2016-05-18 01:47:58,158 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:58,166 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AEB062708FEB4D7E935B1187B1C9AA1F, ordinal: 0
2016-05-18 01:47:58,166 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AEB062708FEB4D7E935B1187B1C9AA1F, mem on start (mb), free: 259, total: 355, max: 672
2016-05-18 01:47:58,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:58,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:58,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:58,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:58,168 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[706EE65833C0453491672AB431EE99BD:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E51CE4596DE1464EA1581641C7F92B05:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:58,169 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:58,169 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:58,170 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:58,170 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[706EE65833C0453491672AB431EE99BD:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E51CE4596DE1464EA1581641C7F92B05:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:58,170 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:58,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 706EE65833C0453491672AB431EE99BD
2016-05-18 01:47:58,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 706EE65833C0453491672AB431EE99BD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:58,172 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:58,172 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:58,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:58,173 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,175 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:58,175 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 706EE65833C0453491672AB431EE99BD initialized RecordReader from event
2016-05-18 01:47:58,177 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:58,178 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 706EE65833C0453491672AB431EE99BD
2016-05-18 01:47:58,178 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 7D480D8D32E140C2A853B2E3039E9DBF
2016-05-18 01:47:58,179 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7D480D8D32E140C2A853B2E3039E9DBF
2016-05-18 01:47:58,182 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E51CE4596DE1464EA1581641C7F92B05 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6ff154c9, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@e25e0d8, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:58,182 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E51CE4596DE1464EA1581641C7F92B05: , UsingHashComparator=false
2016-05-18 01:47:58,195 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:58,195 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:58,196 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E51CE4596DE1464EA1581641C7F92B05: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:58,196 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E51CE4596DE1464EA1581641C7F92B05: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:58,197 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:58,197 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AEB062708FEB4D7E935B1187B1C9AA1F, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:58,203 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,204 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AEB062708FEB4D7E935B1187B1C9AA1F, mem on close (mb), free: 157, total: 355, max: 672
2016-05-18 01:47:58,204 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E51CE4596DE1464EA1581641C7F92B05: Starting flush of map output
2016-05-18 01:47:58,204 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E51CE4596DE1464EA1581641C7F92B05: Span0.length = 37, perItem = 24
2016-05-18 01:47:58,207 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E51CE4596DE1464EA1581641C7F92B05: done sorting span=0, length=37, time=3
2016-05-18 01:47:58,207 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E51CE4596DE1464EA1581641C7F92B05: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:47:58,209 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E51CE4596DE1464EA1581641C7F92B05: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-local-dir/output/attempt_1463536077934_0001_1_00_000000_0_10089_0/file.out
2016-05-18 01:47:58,213 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:58 Completed running task attempt: attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,214 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:58,214 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536077934_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:58,215 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536077934_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536078197, Process_Duration=6, Process_End_Time=1463536078203, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:58,216 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536077934_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,216 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,216 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:58,216 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, taskAttemptId=attempt_1463536077934_0001_1_00_000000_0, creationTime=1463536078139, allocationTime=1463536078144, startTime=1463536078146, finishTime=1463536078216, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:58,217 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077934_0001_00_000001
2016-05-18 01:47:58,217 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536077934_0001_1_00_000000_0
2016-05-18 01:47:58,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_FINISHED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, taskId=task_1463536077934_0001_1_00_000000, startTime=1463536078146, finishTime=1463536078217, timeTaken=71, status=SUCCEEDED, successfulAttemptID=attempt_1463536077934_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536078197, Process_Duration=6, Process_End_Time=1463536078203, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:58,218 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:58,218 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_FINISHED]: vertexName=AEB062708FEB4D7E935B1187B1C9AA1F, vertexId=vertex_1463536077934_0001_1_00, initRequestedTime=1463536078115, initedTime=1463536078133, startRequestedTime=1463536078117, startedTime=1463536078134, finishTime=1463536078218, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536078197, Process_Duration=6, Process_End_Time=1463536078203, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536078146, firstTasksToStart=[ task_1463536077934_0001_1_00_000000 ], lastTaskFinishTime=1463536078216, lastTasksToFinish=[ task_1463536077934_0001_1_00_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536077934_0001_1_00_000000 ], longestDurationTasks=[ task_1463536077934_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:58,219 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:58,220 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] attempt: attempt_1463536077934_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:58,220 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E51CE4596DE1464EA1581641C7F92B05
2016-05-18 01:47:58,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536077934_0001_00_000001, stoppedTime=1463536078220, exitStatus=0
2016-05-18 01:47:58,220 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AEB062708FEB4D7E935B1187B1C9AA1F, destinationVertexName=E51CE4596DE1464EA1581641C7F92B05. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:58,220 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:58,220 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,220 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,220 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536077934_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:58,221 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_STARTED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, taskId=task_1463536077934_0001_1_01_000000, scheduledTime=1463536078221, launchTime=1463536078221
2016-05-18 01:47:58,221 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,223 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536077934_0001_00_000002, launchTime=1463536078222
2016-05-18 01:47:58,223 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077934_0001_00_000002
2016-05-18 01:47:58,224 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077934_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,225 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536077934_0001_1_01_000000_0] started. Is using containerId: [container_1463536077934_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:58 Starting to run new task attempt: attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,225 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0, startTime=1463536078224, containerId=container_1463536077934_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:58,225 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,226 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,226 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:58,226 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false, VertexName: E51CE4596DE1464EA1581641C7F92B05, VertexParallelism: 1, TaskAttemptID:attempt_1463536077934_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AEB062708FEB4D7E935B1187B1C9AA1F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8C16FF468DF742CC81461C81E1B223B7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:58,227 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:58,228 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536077934_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:58,228 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536077934_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,228 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,228 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077934_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:58,228 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536077934_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:58,229 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536077934_0001_00_000001, Exiting
2016-05-18 01:47:58,229 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536077934_0001_00_000001
2016-05-18 01:47:58,229 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536077934_0001_00_000001
2016-05-18 01:47:58,229 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:58,230 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:58,231 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:58,232 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536077934_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:58,232 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536077934_0001_00_000001 completed successfully
2016-05-18 01:47:58,235 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:58,241 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E51CE4596DE1464EA1581641C7F92B05, ordinal: 1
2016-05-18 01:47:58,241 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E51CE4596DE1464EA1581641C7F92B05, mem on start (mb), free: 150, total: 355, max: 672
2016-05-18 01:47:58,241 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:58,242 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:58,242 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:58,244 INFO output.MROutput (MROutput.java:initialize(421)) - 8C16FF468DF742CC81461C81E1B223B7: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:58,245 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:58,245 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AEB062708FEB4D7E935B1187B1C9AA1F:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [8C16FF468DF742CC81461C81E1B223B7:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:58,245 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:58,245 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:58,245 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:58,245 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AEB062708FEB4D7E935B1187B1C9AA1F:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [8C16FF468DF742CC81461C81E1B223B7:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:58,245 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:58,245 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AEB062708FEB4D7E935B1187B1C9AA1F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:58,245 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:58,246 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:58,246 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AEB062708FEB4D7E935B1187B1C9AA1F: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:58,247 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AEB062708FEB4D7E935B1187B1C9AA1F: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:58,247 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AEB062708FEB4D7E935B1187B1C9AA1F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:58,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AEB062708FEB4D7E935B1187B1C9AA1F
2016-05-18 01:47:58,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:58,252 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,252 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AEB062708FEB4D7E935B1187B1C9AA1F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:58,254 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536077934_0001_1_00_000000_0_10089} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536078253, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:47:58,254 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AEB062708FEB4D7E935B1187B1C9AA1F
2016-05-18 01:47:58,254 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:58,254 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AEB062708FEB4D7E935B1187B1C9AA1F, wasInterrupted=false
2016-05-18 01:47:58,255 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:58,255 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:58,256 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:58,256 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 7D480D8D32E140C2A853B2E3039E9DBF
2016-05-18 01:47:58,256 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false"], id: 8C16FF468DF742CC81461C81E1B223B7
2016-05-18 01:47:58,256 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false"]
2016-05-18 01:47:58,257 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7D480D8D32E140C2A853B2E3039E9DBF, for 1 inputs
2016-05-18 01:47:58,257 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001_wd/localmode-local-dir/output/attempt_1463536077934_0001_1_00_000000_0_10089/file.out, len=732, isLocal=true
2016-05-18 01:47:58,257 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:47:58,257 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:58,258 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:58,258 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:47:58,258 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AEB062708FEB4D7E935B1187B1C9AA1F
2016-05-18 01:47:58,258 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E51CE4596DE1464EA1581641C7F92B05, all 1 inputs ready in: 00:00:00.001
2016-05-18 01:47:58,259 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AEB062708FEB4D7E935B1187B1C9AA1F: Shuffle Runner thread complete
2016-05-18 01:47:58,261 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AEB062708FEB4D7E935B1187B1C9AA1F: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:58,265 INFO output.MROutput (MROutput.java:close(526)) - 8C16FF468DF742CC81461C81E1B223B7 closed
2016-05-18 01:47:58,266 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536077934_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:58,267 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,267 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E51CE4596DE1464EA1581641C7F92B05, mem on close (mb), free: 145, total: 355, max: 672
2016-05-18 01:47:58,267 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AEB062708FEB4D7E935B1187B1C9AA1F
2016-05-18 01:47:58,267 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AEB062708FEB4D7E935B1187B1C9AA1F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:58,267 INFO output.MROutput (MROutput.java:close(526)) - 8C16FF468DF742CC81461C81E1B223B7 closed
2016-05-18 01:47:58,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:58 Completed running task attempt: attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,268 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:58,268 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536077934_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:58,269 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536077934_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536078259, Process_Duration=7, Process_End_Time=1463536078266, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:58,270 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536077934_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, taskAttemptId=attempt_1463536077934_0001_1_01_000000_0, creationTime=1463536078221, allocationTime=1463536078222, startTime=1463536078224, finishTime=1463536078270, timeTaken=46, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:58,270 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,270 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:58,270 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,270 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536077934_0001_1_01_000000_0
2016-05-18 01:47:58,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:TASK_FINISHED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, taskId=task_1463536077934_0001_1_01_000000, startTime=1463536078224, finishTime=1463536078270, timeTaken=46, status=SUCCEEDED, successfulAttemptID=attempt_1463536077934_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536078259, Process_Duration=7, Process_End_Time=1463536078266, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:58,271 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:58,271 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05]
2016-05-18 01:47:58,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:VERTEX_FINISHED]: vertexName=E51CE4596DE1464EA1581641C7F92B05, vertexId=vertex_1463536077934_0001_1_01, initRequestedTime=1463536078117, initedTime=1463536078123, startRequestedTime=1463536078135, startedTime=1463536078135, finishTime=1463536078271, timeTaken=136, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536078259, Process_Duration=7, Process_End_Time=1463536078266, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536078224, firstTasksToStart=[ task_1463536077934_0001_1_01_000000 ], lastTaskFinishTime=1463536078270, lastTasksToFinish=[ task_1463536077934_0001_1_01_000000 ], minTaskDuration=46, maxTaskDuration=46, avgTaskDuration=46.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536077934_0001_1_01_000000 ], longestDurationTasks=[ task_1463536077934_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:58,272 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:58,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536077934_0001_00_000002, stoppedTime=1463536078273, exitStatus=0
2016-05-18 01:47:58,273 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:58,273 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,273 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536077934_0001_1_00 [AEB062708FEB4D7E935B1187B1C9AA1F]
2016-05-18 01:47:58,273 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536077934_0001_1
2016-05-18 01:47:58,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536077934_0001_1
2016-05-18 01:47:58,274 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077934_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:58,274 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 8C16FF468DF742CC81461C81E1B223B7 for vertex: vertex_1463536077934_0001_1_01 [E51CE4596DE1464EA1581641C7F92B05], outputName: 8C16FF468DF742CC81461C81E1B223B7
2016-05-18 01:47:58,274 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536077934_0001_00_000002
2016-05-18 01:47:58,274 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536077934_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:58,274 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536077934_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:58,275 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536077934_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:58,275 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536077934_0001_00_000002, Exiting
2016-05-18 01:47:58,275 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536077934_0001_00_000002
2016-05-18 01:47:58,275 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536077934_0001_00_000002
2016-05-18 01:47:58,276 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536077934_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:58,276 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536077934_0001_00_000002 completed successfully
2016-05-18 01:47:58,286 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:8C16FF468DF742CC81461C81E1B223B7 of vertex/vertexGroup:E51CE4596DE1464EA1581641C7F92B05 isVertexGroupOutput:false
2016-05-18 01:47:58,286 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,288 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536077934_0001_1, queueSize=0
2016-05-18 01:47:58,289 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536077934_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536077934_0001_1, startTime=1463536078114, finishTime=1463536078286, timeTaken=172, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=350, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072156456, Process_Duration=13, Process_End_Time=2927072156469, Read_Duration=2, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:58,289 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536077934_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:58,289 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536077934_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:58 Completed Dag: dag_1463536077934_0001_1
2016-05-18 01:47:58,289 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536077934_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:58,289 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536077934_0001_1_post
2016-05-18 01:47:58,289 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,289 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:58,290 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:58,290 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false, with id=dag_1463536077934_0001_1
2016-05-18 01:47:58,290 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62] comprehensivesort/(1/1) ...ive/false_true_true_false, with id=dag_1463536077934_0001_1
2016-05-18 01:47:58,302 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/false_true_true_false, applicationId=application_1463536077934_0001
2016-05-18 01:47:58,304 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:58,304 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:58,304 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:58,304 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:58,304 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:58,304 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:58,305 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:58,306 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:58,306 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:58,306 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:58,307 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:58,307 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:58,307 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:58,307 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:58,311 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/8E36489C9D694E49B4D937E24972EBBB/B04E0BDB3A694081BA73B38F0DED5F62/.tez/application_1463536077934_0001
2016-05-18 01:47:58,311 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false/_temporary
2016-05-18 01:47:58,312 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.406
2016-05-18 01:47:58,382 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:58,382 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:58,425 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 01:47:58,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 01:47:58,427 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:58,427 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:58,427 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:58,451 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:58,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true"]
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:58,556 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:58,559 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/false_null_false_true
2016-05-18 01:47:58,559 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:58,573 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:58,577 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:58,578 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:58,578 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:58,587 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001 doesn't exist and is created
2016-05-18 01:47:58,610 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:58,610 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:58,611 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd
2016-05-18 01:47:58,629 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536078578_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:58,629 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:58,629 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:58,631 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:58,632 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:58,632 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:58,632 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:58,633 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:58,635 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:58,635 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:58,636 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:58,636 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:58,636 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:58,636 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:58,637 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:58,638 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:58,638 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:58,638 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:58,638 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:58,640 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:58,641 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:58,642 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:58,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:58,643 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-log-dir
2016-05-18 01:47:58,643 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-log-dir/history.txt.appattempt_1463536078578_0001_000000, maxErrors=10
2016-05-18 01:47:58,645 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:58,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536078578_0001_000000, appSubmitTime=1463536078628, launchTime=1463536078628
2016-05-18 01:47:58,650 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:50958
2016-05-18 01:47:58,650 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:58,650 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536078578
2016-05-18 01:47:58,653 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536078578_0001_000000, startTime=1463536078653
2016-05-18 01:47:58,653 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:58,710 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:58,711 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:58,711 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/false_null_false_true, applicationId=application_1463536078578_0001, dagName=[50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true
2016-05-18 01:47:58,728 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true
2016-05-18 01:47:58,729 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:58,730 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536078578_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-log-dir/dag_1463536078578_0001_1.dot
2016-05-18 01:47:58,730 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-log-dir/dag_1463536078578_0001_1-tez-dag.pb.txt
2016-05-18 01:47:58,736 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536078578_0001_1
2016-05-18 01:47:58,736 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,737 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true
2016-05-18 01:47:58 Running Dag: dag_1463536078578_0001_1
2016-05-18 01:47:58,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536078578_0001_1, submitTime=1463536078729
2016-05-18 01:47:58,752 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001/tez-conf.pb" } size: 40937 timestamp: 1463536078000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:58,752 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,752 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:58,754 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001/tez-conf.pb" } size: 40937 timestamp: 1463536078000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:58,754 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,755 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:58,755 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:58,755 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536078578_0001_1, initTime=1463536078755
2016-05-18 01:47:58,755 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536078578_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:58,756 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:58,756 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:DAG_STARTED]: dagID=dag_1463536078578_0001_1, startTime=1463536078756
2016-05-18 01:47:58,756 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536078578_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:58,756 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D498B2EB73B74FC7A49FC83CA0CF4486 : {46EB0F81B63C4EEE9878B2E4575379F3={InputName=46EB0F81B63C4EEE9878B2E4575379F3}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:58,756 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 46EB0F81B63C4EEE9878B2E4575379F3, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:58,756 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,757 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,757 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,757 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,758 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/false_null_false_true, applicationId=application_1463536078578_0001, dagName=[50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true
2016-05-18 01:47:58,758 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:58,758 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 46EB0F81B63C4EEE9878B2E4575379F3 on vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,758 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,760 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 46EB0F81B63C4EEE9878B2E4575379F3 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:58,761 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:58,762 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:58,762 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:58,762 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:58,762 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 46EB0F81B63C4EEE9878B2E4575379F3 on vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,762 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:58,763 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,763 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536078578_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:58,763 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640], numAdditionalOutputs=1
2016-05-18 01:47:58,763 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=AFB5844A24E24FD18EE7685C2F50C36C, vertex=vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:58,766 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E42C5B7863014D70A063185A8E71D640:AFB5844A24E24FD18EE7685C2F50C36C using oldmapred API
2016-05-18 01:47:58,766 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=AFB5844A24E24FD18EE7685C2F50C36C, vertexName=E42C5B7863014D70A063185A8E71D640, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:58,767 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E42C5B7863014D70A063185A8E71D640, vertexId=vertex_1463536078578_0001_1_01, initRequestedTime=1463536078758, initedTime=1463536078763, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:58,767 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:58,768 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:58,769 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {46EB0F81B63C4EEE9878B2E4575379F3=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:58,769 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] parallelism set to 1
2016-05-18 01:47:58,769 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:58,770 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, vertexId=vertex_1463536078578_0001_1_00, initRequestedTime=1463536078756, initedTime=1463536078770, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:58,770 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] with distanceFromRoot: 0
2016-05-18 01:47:58,770 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:58,770 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536078578_0001_1_00, startRequestedTime=1463536078758, startedTime=1463536078770
2016-05-18 01:47:58,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536078578_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:58,771 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:58,771 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536078578_0001_1_00 for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] numStartedSources: 1 numSources: 1
2016-05-18 01:47:58,771 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] with distanceFromRoot: 1
2016-05-18 01:47:58,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536078578_0001_1_01, startRequestedTime=1463536078771, startedTime=1463536078771
2016-05-18 01:47:58,771 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:58,773 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D498B2EB73B74FC7A49FC83CA0CF4486
2016-05-18 01:47:58,778 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,778 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E42C5B7863014D70A063185A8E71D640 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:58,779 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_STARTED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, taskId=task_1463536078578_0001_1_00_000000, scheduledTime=1463536078778, launchTime=1463536078778
2016-05-18 01:47:58,779 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,780 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:58,780 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D498B2EB73B74FC7A49FC83CA0CF4486 in vertex: E42C5B7863014D70A063185A8E71D640 numBipartiteSourceTasks: 1
2016-05-18 01:47:58,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536078578_0001_00_000001, launchTime=1463536078781
2016-05-18 01:47:58,782 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536078578_0001_00_000001
2016-05-18 01:47:58,783 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536078578_0001_1_00_000000_0] started. Is using containerId: [container_1463536078578_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:58,784 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0, startTime=1463536078783, containerId=container_1463536078578_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:58,784 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536078578_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58 Starting to run new task attempt: attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,784 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,785 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,785 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:58,785 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536078578_0001, with dag id: dag_1463536078578_0001_1
2016-05-18 01:47:58,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true, VertexName: D498B2EB73B74FC7A49FC83CA0CF4486, VertexParallelism: 1, TaskAttemptID:attempt_1463536078578_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=46EB0F81B63C4EEE9878B2E4575379F3, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E42C5B7863014D70A063185A8E71D640, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:58,786 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:58,786 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,787 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536078578_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:58,787 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536078578_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:58,791 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:58,791 INFO input.MRInput (MRInput.java:initialize(440)) - 46EB0F81B63C4EEE9878B2E4575379F3 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:58,791 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 46EB0F81B63C4EEE9878B2E4575379F3
2016-05-18 01:47:58,797 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D498B2EB73B74FC7A49FC83CA0CF4486, ordinal: 0
2016-05-18 01:47:58,797 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D498B2EB73B74FC7A49FC83CA0CF4486, mem on start (mb), free: 248, total: 352, max: 672
2016-05-18 01:47:58,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:58,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:58,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:58,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:58,798 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[46EB0F81B63C4EEE9878B2E4575379F3:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E42C5B7863014D70A063185A8E71D640:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:58,798 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:58,798 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:58,799 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:58,799 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[46EB0F81B63C4EEE9878B2E4575379F3:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E42C5B7863014D70A063185A8E71D640:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:58,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:58,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 46EB0F81B63C4EEE9878B2E4575379F3 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:58,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:58,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:58,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 46EB0F81B63C4EEE9878B2E4575379F3
2016-05-18 01:47:58,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:58,801 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,804 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:58,805 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:58,806 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 46EB0F81B63C4EEE9878B2E4575379F3 initialized RecordReader from event
2016-05-18 01:47:58,805 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 46EB0F81B63C4EEE9878B2E4575379F3
2016-05-18 01:47:58,806 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 97D96A853A6C4F02939355A6DACB6303
2016-05-18 01:47:58,806 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 97D96A853A6C4F02939355A6DACB6303
2016-05-18 01:47:58,807 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E42C5B7863014D70A063185A8E71D640 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@289f1ca0, comparator=cascading.tuple.hadoop.util.TupleComparator@66106135, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:58,808 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E42C5B7863014D70A063185A8E71D640: , UsingHashComparator=false
2016-05-18 01:47:58,820 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:58,820 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:58,821 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E42C5B7863014D70A063185A8E71D640: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:58,821 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E42C5B7863014D70A063185A8E71D640: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:58,821 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:58,821 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D498B2EB73B74FC7A49FC83CA0CF4486, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:58,825 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,825 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D498B2EB73B74FC7A49FC83CA0CF4486, mem on close (mb), free: 145, total: 352, max: 672
2016-05-18 01:47:58,825 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E42C5B7863014D70A063185A8E71D640: Starting flush of map output
2016-05-18 01:47:58,825 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E42C5B7863014D70A063185A8E71D640: Span0.length = 37, perItem = 17
2016-05-18 01:47:58,826 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E42C5B7863014D70A063185A8E71D640: done sorting span=0, length=37, time=1
2016-05-18 01:47:58,826 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E42C5B7863014D70A063185A8E71D640: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:58,827 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E42C5B7863014D70A063185A8E71D640: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-local-dir/output/attempt_1463536078578_0001_1_00_000000_0_10095_0/file.out
2016-05-18 01:47:58,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:58 Completed running task attempt: attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,830 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:58,830 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536078578_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:58,831 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536078578_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536078821, Process_Duration=4, Process_End_Time=1463536078825, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:58,832 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536078578_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,832 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, taskAttemptId=attempt_1463536078578_0001_1_00_000000_0, creationTime=1463536078779, allocationTime=1463536078781, startTime=1463536078783, finishTime=1463536078832, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:58,832 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,832 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:58,832 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,832 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536078578_0001_1_00_000000_0
2016-05-18 01:47:58,832 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536078578_0001_00_000001
2016-05-18 01:47:58,833 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_FINISHED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, taskId=task_1463536078578_0001_1_00_000000, startTime=1463536078783, finishTime=1463536078832, timeTaken=49, status=SUCCEEDED, successfulAttemptID=attempt_1463536078578_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536078821, Process_Duration=4, Process_End_Time=1463536078825, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:58,833 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:58,834 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,834 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_FINISHED]: vertexName=D498B2EB73B74FC7A49FC83CA0CF4486, vertexId=vertex_1463536078578_0001_1_00, initRequestedTime=1463536078756, initedTime=1463536078770, startRequestedTime=1463536078758, startedTime=1463536078770, finishTime=1463536078834, timeTaken=64, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536078821, Process_Duration=4, Process_End_Time=1463536078825, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536078783, firstTasksToStart=[ task_1463536078578_0001_1_00_000000 ], lastTaskFinishTime=1463536078832, lastTasksToFinish=[ task_1463536078578_0001_1_00_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536078578_0001_1_00_000000 ], longestDurationTasks=[ task_1463536078578_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:58,834 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:58,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536078578_0001_00_000001, stoppedTime=1463536078835, exitStatus=0
2016-05-18 01:47:58,835 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] attempt: attempt_1463536078578_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:58,835 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:58,835 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,835 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E42C5B7863014D70A063185A8E71D640
2016-05-18 01:47:58,835 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D498B2EB73B74FC7A49FC83CA0CF4486, destinationVertexName=E42C5B7863014D70A063185A8E71D640. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:58,835 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,836 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536078578_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:58,836 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_STARTED]: vertexName=E42C5B7863014D70A063185A8E71D640, taskId=task_1463536078578_0001_1_01_000000, scheduledTime=1463536078836, launchTime=1463536078836
2016-05-18 01:47:58,836 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536078578_0001_00_000002, launchTime=1463536078838
2016-05-18 01:47:58,838 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536078578_0001_00_000002
2016-05-18 01:47:58,839 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536078578_0001_1_01_000000_0] started. Is using containerId: [container_1463536078578_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:58,839 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E42C5B7863014D70A063185A8E71D640, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0, startTime=1463536078839, containerId=container_1463536078578_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:58,839 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536078578_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58 Starting to run new task attempt: attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,840 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,840 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,840 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:58,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true, VertexName: E42C5B7863014D70A063185A8E71D640, VertexParallelism: 1, TaskAttemptID:attempt_1463536078578_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D498B2EB73B74FC7A49FC83CA0CF4486, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AFB5844A24E24FD18EE7685C2F50C36C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:58,840 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:58,841 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,841 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536078578_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,841 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536078578_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:58,844 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536078578_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:58,844 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536078578_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:58,844 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536078578_0001_00_000001, Exiting
2016-05-18 01:47:58,844 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536078578_0001_00_000001
2016-05-18 01:47:58,844 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536078578_0001_00_000001
2016-05-18 01:47:58,845 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:58,846 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536078578_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:58,847 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536078578_0001_00_000001 completed successfully
2016-05-18 01:47:58,852 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E42C5B7863014D70A063185A8E71D640, ordinal: 1
2016-05-18 01:47:58,852 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E42C5B7863014D70A063185A8E71D640, mem on start (mb), free: 141, total: 352, max: 672
2016-05-18 01:47:58,852 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:58,852 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:58,854 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:58,857 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:58,858 INFO output.MROutput (MROutput.java:initialize(421)) - AFB5844A24E24FD18EE7685C2F50C36C: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:58,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:58,858 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AFB5844A24E24FD18EE7685C2F50C36C:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D498B2EB73B74FC7A49FC83CA0CF4486:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:58,859 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:58,859 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:58,859 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:58,859 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AFB5844A24E24FD18EE7685C2F50C36C:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D498B2EB73B74FC7A49FC83CA0CF4486:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:58,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:58,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D498B2EB73B74FC7A49FC83CA0CF4486 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:58,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:58,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:58,860 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D498B2EB73B74FC7A49FC83CA0CF4486: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:58,861 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D498B2EB73B74FC7A49FC83CA0CF4486: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:58,862 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D498B2EB73B74FC7A49FC83CA0CF4486 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:58,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D498B2EB73B74FC7A49FC83CA0CF4486
2016-05-18 01:47:58,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:58,865 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,867 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D498B2EB73B74FC7A49FC83CA0CF4486: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:58,869 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:58,869 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 97D96A853A6C4F02939355A6DACB6303
2016-05-18 01:47:58,869 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true"], id: AFB5844A24E24FD18EE7685C2F50C36C
2016-05-18 01:47:58,870 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true"]
2016-05-18 01:47:58,870 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 97D96A853A6C4F02939355A6DACB6303, for 1 inputs
2016-05-18 01:47:58,871 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536078578_0001_1_00_000000_0_10095} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536078871, TimeTaken=1, Rate=0.57 MB/s
2016-05-18 01:47:58,871 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D498B2EB73B74FC7A49FC83CA0CF4486
2016-05-18 01:47:58,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:58,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D498B2EB73B74FC7A49FC83CA0CF4486, wasInterrupted=false
2016-05-18 01:47:58,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - D498B2EB73B74FC7A49FC83CA0CF4486: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:58,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:58,873 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:58,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001_wd/localmode-local-dir/output/attempt_1463536078578_0001_1_00_000000_0_10095/file.out, len=593, isLocal=true
2016-05-18 01:47:58,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:58,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:58,874 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:58,874 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:58,874 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D498B2EB73B74FC7A49FC83CA0CF4486
2016-05-18 01:47:58,874 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D498B2EB73B74FC7A49FC83CA0CF4486: Shuffle Runner thread complete
2016-05-18 01:47:58,875 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E42C5B7863014D70A063185A8E71D640, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:47:58,876 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D498B2EB73B74FC7A49FC83CA0CF4486: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:58,883 INFO output.MROutput (MROutput.java:close(526)) - AFB5844A24E24FD18EE7685C2F50C36C closed
2016-05-18 01:47:58,884 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536078578_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:58,884 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,885 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E42C5B7863014D70A063185A8E71D640, mem on close (mb), free: 135, total: 352, max: 672
2016-05-18 01:47:58,885 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D498B2EB73B74FC7A49FC83CA0CF4486
2016-05-18 01:47:58,885 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D498B2EB73B74FC7A49FC83CA0CF4486: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:58,885 INFO output.MROutput (MROutput.java:close(526)) - AFB5844A24E24FD18EE7685C2F50C36C closed
2016-05-18 01:47:58,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:58 Completed running task attempt: attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,886 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:58,887 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536078578_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:58,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536078578_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536078875, Process_Duration=9, Process_End_Time=1463536078884, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:58,888 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536078578_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,889 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:58,889 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:58,889 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E42C5B7863014D70A063185A8E71D640, taskAttemptId=attempt_1463536078578_0001_1_01_000000_0, creationTime=1463536078836, allocationTime=1463536078837, startTime=1463536078839, finishTime=1463536078889, timeTaken=50, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:58,889 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536078578_0001_00_000002
2016-05-18 01:47:58,889 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536078578_0001_1_01_000000_0
2016-05-18 01:47:58,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:TASK_FINISHED]: vertexName=E42C5B7863014D70A063185A8E71D640, taskId=task_1463536078578_0001_1_01_000000, startTime=1463536078839, finishTime=1463536078889, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463536078578_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536078875, Process_Duration=9, Process_End_Time=1463536078884, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:58,890 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:58,890 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640]
2016-05-18 01:47:58,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:VERTEX_FINISHED]: vertexName=E42C5B7863014D70A063185A8E71D640, vertexId=vertex_1463536078578_0001_1_01, initRequestedTime=1463536078758, initedTime=1463536078763, startRequestedTime=1463536078771, startedTime=1463536078771, finishTime=1463536078890, timeTaken=119, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536078875, Process_Duration=9, Process_End_Time=1463536078884, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536078839, firstTasksToStart=[ task_1463536078578_0001_1_01_000000 ], lastTaskFinishTime=1463536078889, lastTasksToFinish=[ task_1463536078578_0001_1_01_000000 ], minTaskDuration=50, maxTaskDuration=50, avgTaskDuration=50.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536078578_0001_1_01_000000 ], longestDurationTasks=[ task_1463536078578_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:58,891 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:58,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536078578_0001_00_000002, stoppedTime=1463536078892, exitStatus=0
2016-05-18 01:47:58,892 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:58,892 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,892 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536078578_0001_1_00 [D498B2EB73B74FC7A49FC83CA0CF4486]
2016-05-18 01:47:58,892 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536078578_0001_1
2016-05-18 01:47:58,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536078578_0001_1
2016-05-18 01:47:58,893 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: AFB5844A24E24FD18EE7685C2F50C36C for vertex: vertex_1463536078578_0001_1_01 [E42C5B7863014D70A063185A8E71D640], outputName: AFB5844A24E24FD18EE7685C2F50C36C
2016-05-18 01:47:58,895 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:58,895 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:58,895 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536078578_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:58,896 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536078578_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:58,900 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536078578_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:58,901 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536078578_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:58,901 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536078578_0001_00_000002, Exiting
2016-05-18 01:47:58,901 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536078578_0001_00_000002
2016-05-18 01:47:58,901 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536078578_0001_00_000002
2016-05-18 01:47:58,901 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536078578_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:58,901 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536078578_0001_00_000002 completed successfully
2016-05-18 01:47:58,905 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:AFB5844A24E24FD18EE7685C2F50C36C of vertex/vertexGroup:E42C5B7863014D70A063185A8E71D640 isVertexGroupOutput:false
2016-05-18 01:47:58,905 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:58,907 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536078578_0001_1, queueSize=0
2016-05-18 01:47:58,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536078578_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536078578_0001_1, startTime=1463536078756, finishTime=1463536078905, timeTaken=149, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072157696, Process_Duration=13, Process_End_Time=2927072157709, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:58,908 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536078578_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:58,908 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536078578_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:58 Completed Dag: dag_1463536078578_0001_1
2016-05-18 01:47:58,908 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536078578_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:58,908 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536078578_0001_1_post
2016-05-18 01:47:58,909 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:58,909 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:58,909 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:58,910 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true, with id=dag_1463536078578_0001_1
2016-05-18 01:47:58,910 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2] comprehensivesort/(1/1) ...ive/false_null_false_true, with id=dag_1463536078578_0001_1
2016-05-18 01:47:58,916 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/false_null_false_true, applicationId=application_1463536078578_0001
2016-05-18 01:47:58,917 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:58,917 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:58,917 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:58,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:58,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:58,918 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:58,918 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:58,919 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:58,920 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:58,920 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:58,920 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:58,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:58,921 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:58,921 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:58,926 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/50718AD6D34448F3B11AC66C274B1B6B/C78A1D0382A4460B9C48070DB85607D2/.tez/application_1463536078578_0001
2016-05-18 01:47:58,927 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true/_temporary
2016-05-18 01:47:58,927 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.370
2016-05-18 01:47:59,008 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:59,009 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:59,068 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 01:47:59,076 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.024
2016-05-18 01:47:59,077 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:59,077 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:59,077 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:59,093 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:59,100 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true
2016-05-18 01:47:59,151 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:59,151 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,152 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true"]
2016-05-18 01:47:59,152 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:59,152 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:59,152 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:59,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/true_null_false_true
2016-05-18 01:47:59,153 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:59,166 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:59,170 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:59,171 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:59,172 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:59,181 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001 doesn't exist and is created
2016-05-18 01:47:59,200 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:59,200 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:59,201 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd
2016-05-18 01:47:59,216 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536079171_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:59,216 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:59,216 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:59,218 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:59,218 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:59,218 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:59,218 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:59,219 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:59,221 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:59,221 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:59,221 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:59,221 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:59,221 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:59,222 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:59,222 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:59,222 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:59,222 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:59,222 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:59,223 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:59,223 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:59,225 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:59,226 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:59,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:59,227 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-log-dir
2016-05-18 01:47:59,227 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-log-dir/history.txt.appattempt_1463536079171_0001_000000, maxErrors=10
2016-05-18 01:47:59,227 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:59,228 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536079171_0001_000000, appSubmitTime=1463536079215, launchTime=1463536079215
2016-05-18 01:47:59,230 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:59,237 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:34644
2016-05-18 01:47:59,238 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536079171
2016-05-18 01:47:59,240 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536079171_0001_000000, startTime=1463536079240
2016-05-18 01:47:59,240 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:59,300 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:59,301 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:59,301 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/true_null_false_true, applicationId=application_1463536079171_0001, dagName=[FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true
2016-05-18 01:47:59,317 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true
2016-05-18 01:47:59,317 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:59,318 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536079171_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-log-dir/dag_1463536079171_0001_1.dot
2016-05-18 01:47:59,319 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-log-dir/dag_1463536079171_0001_1-tez-dag.pb.txt
2016-05-18 01:47:59,328 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536079171_0001_1
2016-05-18 01:47:59,328 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,328 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true
2016-05-18 01:47:59 Running Dag: dag_1463536079171_0001_1
2016-05-18 01:47:59,343 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536079171_0001_1, submitTime=1463536079317
2016-05-18 01:47:59,344 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001/tez-conf.pb" } size: 40937 timestamp: 1463536079000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:59,344 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,344 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:59,345 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001/tez-conf.pb" } size: 40937 timestamp: 1463536079000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:59,345 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,345 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:59,345 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:59,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536079171_0001_1, initTime=1463536079346
2016-05-18 01:47:59,346 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079171_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:59,347 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:59,347 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:DAG_STARTED]: dagID=dag_1463536079171_0001_1, startTime=1463536079347
2016-05-18 01:47:59,347 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079171_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:59,347 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: FE8760FF91D84CDC9846834DD4D32FFA : {02D51033477A4459813A89421AB41D39={InputName=02D51033477A4459813A89421AB41D39}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:59,347 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 02D51033477A4459813A89421AB41D39, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:59,348 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,348 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,348 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,348 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,349 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:59,349 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,349 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 02D51033477A4459813A89421AB41D39 on vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,352 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 02D51033477A4459813A89421AB41D39 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:59,354 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/true_null_false_true, applicationId=application_1463536079171_0001, dagName=[FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true
2016-05-18 01:47:59,354 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:59,355 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,355 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,355 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536079171_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:59,355 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF], numAdditionalOutputs=1
2016-05-18 01:47:59,356 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=46C617CFE2A3474B8DDF46FC9652293D, vertex=vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:59,357 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:59,357 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:59,357 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:59,358 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:59,358 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 02D51033477A4459813A89421AB41D39 on vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,359 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F346C421447E4348AE196B4FCB5D82DF:46C617CFE2A3474B8DDF46FC9652293D using oldmapred API
2016-05-18 01:47:59,359 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=46C617CFE2A3474B8DDF46FC9652293D, vertexName=F346C421447E4348AE196B4FCB5D82DF, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:59,360 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, vertexId=vertex_1463536079171_0001_1_01, initRequestedTime=1463536079349, initedTime=1463536079355, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:59,360 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:59,361 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:59,361 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {02D51033477A4459813A89421AB41D39=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:59,362 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] parallelism set to 1
2016-05-18 01:47:59,362 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:59,363 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, vertexId=vertex_1463536079171_0001_1_00, initRequestedTime=1463536079347, initedTime=1463536079362, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:59,363 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] with distanceFromRoot: 0
2016-05-18 01:47:59,363 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:59,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536079171_0001_1_00, startRequestedTime=1463536079349, startedTime=1463536079363
2016-05-18 01:47:59,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536079171_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:59,364 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:59,364 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536079171_0001_1_00 for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] numStartedSources: 1 numSources: 1
2016-05-18 01:47:59,364 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] with distanceFromRoot: 1
2016-05-18 01:47:59,365 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in FE8760FF91D84CDC9846834DD4D32FFA
2016-05-18 01:47:59,365 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,365 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536079171_0001_1_01, startRequestedTime=1463536079364, startedTime=1463536079365
2016-05-18 01:47:59,367 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:59,368 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F346C421447E4348AE196B4FCB5D82DF with 0 source tasks and 1 pending tasks
2016-05-18 01:47:59,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_STARTED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, taskId=task_1463536079171_0001_1_00_000000, scheduledTime=1463536079368, launchTime=1463536079368
2016-05-18 01:47:59,369 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,370 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: FE8760FF91D84CDC9846834DD4D32FFA in vertex: F346C421447E4348AE196B4FCB5D82DF numBipartiteSourceTasks: 1
2016-05-18 01:47:59,371 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:59,372 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536079171_0001_00_000001, launchTime=1463536079372
2016-05-18 01:47:59,373 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079171_0001_00_000001
2016-05-18 01:47:59,374 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079171_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,374 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536079171_0001_1_00_000000_0] started. Is using containerId: [container_1463536079171_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:59 Starting to run new task attempt: attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,374 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0, startTime=1463536079374, containerId=container_1463536079171_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:59,374 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,375 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,375 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:59,375 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true, VertexName: FE8760FF91D84CDC9846834DD4D32FFA, VertexParallelism: 1, TaskAttemptID:attempt_1463536079171_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=02D51033477A4459813A89421AB41D39, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F346C421447E4348AE196B4FCB5D82DF, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:59,375 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:59,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536079171_0001, with dag id: dag_1463536079171_0001_1
2016-05-18 01:47:59,376 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,380 INFO input.MRInput (MRInput.java:initialize(440)) - 02D51033477A4459813A89421AB41D39 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:59,380 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 02D51033477A4459813A89421AB41D39
2016-05-18 01:47:59,380 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:59,381 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536079171_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:59,381 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536079171_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:59,387 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FE8760FF91D84CDC9846834DD4D32FFA, ordinal: 0
2016-05-18 01:47:59,387 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FE8760FF91D84CDC9846834DD4D32FFA, mem on start (mb), free: 233, total: 352, max: 672
2016-05-18 01:47:59,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:59,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:59,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:59,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:59,389 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[02D51033477A4459813A89421AB41D39:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F346C421447E4348AE196B4FCB5D82DF:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:59,389 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:59,389 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:59,390 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:59,390 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[02D51033477A4459813A89421AB41D39:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F346C421447E4348AE196B4FCB5D82DF:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:59,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:59,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 02D51033477A4459813A89421AB41D39 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:59,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:59,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:59,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 02D51033477A4459813A89421AB41D39
2016-05-18 01:47:59,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:59,392 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,393 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:59,393 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 02D51033477A4459813A89421AB41D39 initialized RecordReader from event
2016-05-18 01:47:59,395 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,396 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 02D51033477A4459813A89421AB41D39
2016-05-18 01:47:59,396 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 7E3E9E84F6D94FCAB2C9CC0F82F764D2
2016-05-18 01:47:59,396 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7E3E9E84F6D94FCAB2C9CC0F82F764D2
2016-05-18 01:47:59,398 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F346C421447E4348AE196B4FCB5D82DF using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@b3db1ee, comparator=cascading.tuple.hadoop.util.TupleComparator@7f6c8605, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:59,399 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F346C421447E4348AE196B4FCB5D82DF: , UsingHashComparator=false
2016-05-18 01:47:59,411 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:59,411 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:59,412 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F346C421447E4348AE196B4FCB5D82DF: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:59,412 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F346C421447E4348AE196B4FCB5D82DF: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:59,412 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,413 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FE8760FF91D84CDC9846834DD4D32FFA, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:59,417 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,417 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FE8760FF91D84CDC9846834DD4D32FFA, mem on close (mb), free: 131, total: 352, max: 672
2016-05-18 01:47:59,417 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F346C421447E4348AE196B4FCB5D82DF: Starting flush of map output
2016-05-18 01:47:59,418 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F346C421447E4348AE196B4FCB5D82DF: Span0.length = 37, perItem = 17
2016-05-18 01:47:59,418 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F346C421447E4348AE196B4FCB5D82DF: done sorting span=0, length=37, time=0
2016-05-18 01:47:59,419 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F346C421447E4348AE196B4FCB5D82DF: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:47:59,420 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F346C421447E4348AE196B4FCB5D82DF: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-local-dir/output/attempt_1463536079171_0001_1_00_000000_0_10101_0/file.out
2016-05-18 01:47:59,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:59 Completed running task attempt: attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,424 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:59,424 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536079171_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:59,425 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536079171_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536079413, Process_Duration=4, Process_End_Time=1463536079417, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:47:59,426 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536079171_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:59,426 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:59,426 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:59,426 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,426 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079171_0001_00_000001
2016-05-18 01:47:59,426 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, taskAttemptId=attempt_1463536079171_0001_1_00_000000_0, creationTime=1463536079369, allocationTime=1463536079372, startTime=1463536079374, finishTime=1463536079426, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:59,427 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536079171_0001_1_00_000000_0
2016-05-18 01:47:59,428 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_FINISHED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, taskId=task_1463536079171_0001_1_00_000000, startTime=1463536079374, finishTime=1463536079427, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463536079171_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536079413, Process_Duration=4, Process_End_Time=1463536079417, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:47:59,428 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:59,428 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,430 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_FINISHED]: vertexName=FE8760FF91D84CDC9846834DD4D32FFA, vertexId=vertex_1463536079171_0001_1_00, initRequestedTime=1463536079347, initedTime=1463536079362, startRequestedTime=1463536079349, startedTime=1463536079363, finishTime=1463536079428, timeTaken=65, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536079413, Process_Duration=4, Process_End_Time=1463536079417, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536079374, firstTasksToStart=[ task_1463536079171_0001_1_00_000000 ], lastTaskFinishTime=1463536079426, lastTasksToFinish=[ task_1463536079171_0001_1_00_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536079171_0001_1_00_000000 ], longestDurationTasks=[ task_1463536079171_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:59,431 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:59,431 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] attempt: attempt_1463536079171_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:59,431 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536079171_0001_00_000001, stoppedTime=1463536079431, exitStatus=0
2016-05-18 01:47:59,431 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:59,431 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:59,431 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536079171_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:59,432 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F346C421447E4348AE196B4FCB5D82DF
2016-05-18 01:47:59,432 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=FE8760FF91D84CDC9846834DD4D32FFA, destinationVertexName=F346C421447E4348AE196B4FCB5D82DF. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:59,432 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,433 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_STARTED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, taskId=task_1463536079171_0001_1_01_000000, scheduledTime=1463536079433, launchTime=1463536079433
2016-05-18 01:47:59,433 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,434 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536079171_0001_00_000002, launchTime=1463536079434
2016-05-18 01:47:59,435 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079171_0001_00_000002
2016-05-18 01:47:59,436 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536079171_0001_1_01_000000_0] started. Is using containerId: [container_1463536079171_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:59,436 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0, startTime=1463536079436, containerId=container_1463536079171_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:59,436 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079171_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59 Starting to run new task attempt: attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,437 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,437 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,437 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:59,437 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true, VertexName: F346C421447E4348AE196B4FCB5D82DF, VertexParallelism: 1, TaskAttemptID:attempt_1463536079171_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=FE8760FF91D84CDC9846834DD4D32FFA, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=46C617CFE2A3474B8DDF46FC9652293D, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:59,438 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:59,438 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,439 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536079171_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,439 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536079171_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:59,439 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079171_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:59,439 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536079171_0001_00_000001, Exiting
2016-05-18 01:47:59,439 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536079171_0001_00_000001
2016-05-18 01:47:59,439 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536079171_0001_00_000001
2016-05-18 01:47:59,439 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536079171_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:59,442 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:59,443 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536079171_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:59,443 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536079171_0001_00_000001 completed successfully
2016-05-18 01:47:59,443 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:59,453 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F346C421447E4348AE196B4FCB5D82DF, ordinal: 1
2016-05-18 01:47:59,453 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F346C421447E4348AE196B4FCB5D82DF, mem on start (mb), free: 126, total: 352, max: 672
2016-05-18 01:47:59,454 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:59,454 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:59,454 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:59,458 INFO output.MROutput (MROutput.java:initialize(421)) - 46C617CFE2A3474B8DDF46FC9652293D: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:59,459 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:59,459 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[FE8760FF91D84CDC9846834DD4D32FFA:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [46C617CFE2A3474B8DDF46FC9652293D:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:59,459 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:59,460 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:59,460 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:59,460 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[FE8760FF91D84CDC9846834DD4D32FFA:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [46C617CFE2A3474B8DDF46FC9652293D:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:59,460 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:59,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: FE8760FF91D84CDC9846834DD4D32FFA being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:59,461 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - FE8760FF91D84CDC9846834DD4D32FFA: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:59,461 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - FE8760FF91D84CDC9846834DD4D32FFA: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:59,462 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: FE8760FF91D84CDC9846834DD4D32FFA with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:59,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:59,463 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:59,464 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: FE8760FF91D84CDC9846834DD4D32FFA
2016-05-18 01:47:59,464 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:59,464 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,465 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - FE8760FF91D84CDC9846834DD4D32FFA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:59,468 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536079171_0001_1_00_000000_0_10101} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536079468, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:47:59,468 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : FE8760FF91D84CDC9846834DD4D32FFA
2016-05-18 01:47:59,468 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:59,468 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:47:59,468 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: FE8760FF91D84CDC9846834DD4D32FFA, wasInterrupted=false
2016-05-18 01:47:59,470 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 7E3E9E84F6D94FCAB2C9CC0F82F764D2
2016-05-18 01:47:59,470 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true"], id: 46C617CFE2A3474B8DDF46FC9652293D
2016-05-18 01:47:59,470 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true"]
2016-05-18 01:47:59,470 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:59,470 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 7E3E9E84F6D94FCAB2C9CC0F82F764D2, for 1 inputs
2016-05-18 01:47:59,472 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:59,472 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001_wd/localmode-local-dir/output/attempt_1463536079171_0001_1_00_000000_0_10101/file.out, len=593, isLocal=true
2016-05-18 01:47:59,473 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:47:59,473 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:59,473 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:59,473 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:47:59,473 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : FE8760FF91D84CDC9846834DD4D32FFA
2016-05-18 01:47:59,473 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - FE8760FF91D84CDC9846834DD4D32FFA: Shuffle Runner thread complete
2016-05-18 01:47:59,474 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F346C421447E4348AE196B4FCB5D82DF, all 1 inputs ready in: 00:00:00.003
2016-05-18 01:47:59,474 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - FE8760FF91D84CDC9846834DD4D32FFA: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:59,479 INFO output.MROutput (MROutput.java:close(526)) - 46C617CFE2A3474B8DDF46FC9652293D closed
2016-05-18 01:47:59,479 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536079171_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:59,480 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,480 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F346C421447E4348AE196B4FCB5D82DF, mem on close (mb), free: 121, total: 352, max: 672
2016-05-18 01:47:59,480 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: FE8760FF91D84CDC9846834DD4D32FFA
2016-05-18 01:47:59,480 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - FE8760FF91D84CDC9846834DD4D32FFA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:59,480 INFO output.MROutput (MROutput.java:close(526)) - 46C617CFE2A3474B8DDF46FC9652293D closed
2016-05-18 01:47:59,481 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:59 Completed running task attempt: attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,481 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:59,482 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536079171_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:59,482 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536079171_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536079474, Process_Duration=6, Process_End_Time=1463536079480, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:47:59,483 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536079171_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:59,483 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, taskAttemptId=attempt_1463536079171_0001_1_01_000000_0, creationTime=1463536079433, allocationTime=1463536079434, startTime=1463536079436, finishTime=1463536079483, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:59,483 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:59,484 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:59,484 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,484 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079171_0001_00_000002
2016-05-18 01:47:59,484 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536079171_0001_1_01_000000_0
2016-05-18 01:47:59,485 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:TASK_FINISHED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, taskId=task_1463536079171_0001_1_01_000000, startTime=1463536079436, finishTime=1463536079484, timeTaken=48, status=SUCCEEDED, successfulAttemptID=attempt_1463536079171_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536079474, Process_Duration=6, Process_End_Time=1463536079480, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:47:59,486 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:59,486 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF]
2016-05-18 01:47:59,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:VERTEX_FINISHED]: vertexName=F346C421447E4348AE196B4FCB5D82DF, vertexId=vertex_1463536079171_0001_1_01, initRequestedTime=1463536079349, initedTime=1463536079355, startRequestedTime=1463536079364, startedTime=1463536079365, finishTime=1463536079486, timeTaken=121, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536079474, Process_Duration=6, Process_End_Time=1463536079480, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536079436, firstTasksToStart=[ task_1463536079171_0001_1_01_000000 ], lastTaskFinishTime=1463536079483, lastTasksToFinish=[ task_1463536079171_0001_1_01_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536079171_0001_1_01_000000 ], longestDurationTasks=[ task_1463536079171_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:59,487 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:59,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536079171_0001_00_000002, stoppedTime=1463536079487, exitStatus=0
2016-05-18 01:47:59,487 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:59,487 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:59,487 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536079171_0001_1_00 [FE8760FF91D84CDC9846834DD4D32FFA]
2016-05-18 01:47:59,487 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536079171_0001_1
2016-05-18 01:47:59,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536079171_0001_1
2016-05-18 01:47:59,488 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079171_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:59,488 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 46C617CFE2A3474B8DDF46FC9652293D for vertex: vertex_1463536079171_0001_1_01 [F346C421447E4348AE196B4FCB5D82DF], outputName: 46C617CFE2A3474B8DDF46FC9652293D
2016-05-18 01:47:59,488 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536079171_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:59,495 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536079171_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:59,495 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079171_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:59,495 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536079171_0001_00_000002, Exiting
2016-05-18 01:47:59,495 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536079171_0001_00_000002
2016-05-18 01:47:59,495 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536079171_0001_00_000002
2016-05-18 01:47:59,495 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536079171_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:59,495 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536079171_0001_00_000002 completed successfully
2016-05-18 01:47:59,497 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:46C617CFE2A3474B8DDF46FC9652293D of vertex/vertexGroup:F346C421447E4348AE196B4FCB5D82DF isVertexGroupOutput:false
2016-05-18 01:47:59,497 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:59,498 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:59,499 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:59,500 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536079171_0001_1, queueSize=0
2016-05-18 01:47:59,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079171_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536079171_0001_1, startTime=1463536079347, finishTime=1463536079497, timeTaken=150, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=250, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072158887, Process_Duration=10, Process_End_Time=2927072158897, Read_Duration=3, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:47:59,500 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536079171_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:59,500 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079171_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:59 Completed Dag: dag_1463536079171_0001_1
2016-05-18 01:47:59,501 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536079171_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:59,501 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536079171_0001_1_post
2016-05-18 01:47:59,501 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,501 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:59,501 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:59,501 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true, with id=dag_1463536079171_0001_1
2016-05-18 01:47:59,501 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E] comprehensivesort/(1/1) ...sive/true_null_false_true, with id=dag_1463536079171_0001_1
2016-05-18 01:47:59,513 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/true_null_false_true, applicationId=application_1463536079171_0001
2016-05-18 01:47:59,515 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:59,515 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:59,515 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:59,515 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:59,515 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:59,516 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:59,516 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:59,517 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:59,517 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:59,517 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:59,517 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:59,517 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:59,517 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:59,518 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:59,522 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FF4C3B8BE76849A0977B7847A817FB6A/C4E087F980DF4451A46B3E21D744835E/.tez/application_1463536079171_0001
2016-05-18 01:47:59,523 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true/_temporary
2016-05-18 01:47:59,523 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.372
2016-05-18 01:47:59,609 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:59,609 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:59,644 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.008
2016-05-18 01:47:59,653 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 01:47:59,653 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:59,654 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:59,655 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:59,676 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:47:59,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true
2016-05-18 01:47:59,734 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:47:59,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true"]
2016-05-18 01:47:59,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:47:59,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:47:59,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:47:59,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/false_null_true_true
2016-05-18 01:47:59,736 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:59,750 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:47:59,754 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:59,755 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:59,755 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:59,763 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001 doesn't exist and is created
2016-05-18 01:47:59,784 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:59,784 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:59,785 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd
2016-05-18 01:47:59,801 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536079754_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:59,801 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:59,801 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:59,803 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:59,803 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:59,803 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:59,803 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:59,803 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:59,805 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:59,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:59,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:59,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:59,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:59,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:59,806 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:59,808 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:59,809 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:59,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:59,810 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-log-dir
2016-05-18 01:47:59,810 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-log-dir/history.txt.appattempt_1463536079754_0001_000000, maxErrors=10
2016-05-18 01:47:59,810 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:59,811 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536079754_0001_000000, appSubmitTime=1463536079800, launchTime=1463536079800
2016-05-18 01:47:59,815 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:59,816 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:34907
2016-05-18 01:47:59,817 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536079754
2016-05-18 01:47:59,821 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536079754_0001_000000, startTime=1463536079821
2016-05-18 01:47:59,821 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:59,885 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:59,885 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:59,885 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/false_null_true_true, applicationId=application_1463536079754_0001, dagName=[10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true
2016-05-18 01:47:59,902 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true
2016-05-18 01:47:59,902 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:59,904 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536079754_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-log-dir/dag_1463536079754_0001_1.dot
2016-05-18 01:47:59,904 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-log-dir/dag_1463536079754_0001_1-tez-dag.pb.txt
2016-05-18 01:47:59,911 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536079754_0001_1
2016-05-18 01:47:59,911 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,911 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true
2016-05-18 01:47:59 Running Dag: dag_1463536079754_0001_1
2016-05-18 01:47:59,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536079754_0001_1, submitTime=1463536079902
2016-05-18 01:47:59,928 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001/tez-conf.pb" } size: 40937 timestamp: 1463536079000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:59,928 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,928 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:59,929 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001/tez-conf.pb" } size: 40937 timestamp: 1463536079000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:59,929 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:47:59,929 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:59,929 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:59,929 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536079754_0001_1, initTime=1463536079929
2016-05-18 01:47:59,930 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079754_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:59,931 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:59,931 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:DAG_STARTED]: dagID=dag_1463536079754_0001_1, startTime=1463536079931
2016-05-18 01:47:59,931 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079754_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:59,931 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: EB455D08F78D45C1938C862CB0322F2B : {AA9D1AAC54FC40ABB121876964D8D2E7={InputName=AA9D1AAC54FC40ABB121876964D8D2E7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:59,931 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: AA9D1AAC54FC40ABB121876964D8D2E7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:59,931 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,931 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,932 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,932 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,933 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:59,934 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:47:59,933 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: AA9D1AAC54FC40ABB121876964D8D2E7 on vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,934 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/false_null_true_true, applicationId=application_1463536079754_0001, dagName=[10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true
2016-05-18 01:47:59,937 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:59,937 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:47:59,937 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input AA9D1AAC54FC40ABB121876964D8D2E7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:59,938 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:47:59,938 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536079754_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:59,939 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888], numAdditionalOutputs=1
2016-05-18 01:47:59,939 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=41D9FE0855424CED8E46CD99EBD51409, vertex=vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:59,940 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:59,940 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:47:59,940 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:59,940 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:59,941 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A18AA6EEE00D48C09D6A5037A757E888:41D9FE0855424CED8E46CD99EBD51409 using oldmapred API
2016-05-18 01:47:59,942 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=41D9FE0855424CED8E46CD99EBD51409, vertexName=A18AA6EEE00D48C09D6A5037A757E888, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:59,943 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: AA9D1AAC54FC40ABB121876964D8D2E7 on vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,943 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, vertexId=vertex_1463536079754_0001_1_01, initRequestedTime=1463536079934, initedTime=1463536079939, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:59,943 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:59,944 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:59,945 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {AA9D1AAC54FC40ABB121876964D8D2E7=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:59,945 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] parallelism set to 1
2016-05-18 01:47:59,945 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:59,946 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, vertexId=vertex_1463536079754_0001_1_00, initRequestedTime=1463536079931, initedTime=1463536079946, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:59,946 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] with distanceFromRoot: 0
2016-05-18 01:47:59,947 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:59,947 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536079754_0001_1_00, startRequestedTime=1463536079933, startedTime=1463536079947
2016-05-18 01:47:59,947 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536079754_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:59,948 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:59,948 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536079754_0001_1_00 for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] numStartedSources: 1 numSources: 1
2016-05-18 01:47:59,948 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] with distanceFromRoot: 1
2016-05-18 01:47:59,948 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in EB455D08F78D45C1938C862CB0322F2B
2016-05-18 01:47:59,949 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,949 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536079754_0001_1_01, startRequestedTime=1463536079948, startedTime=1463536079948
2016-05-18 01:47:59,949 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:59,951 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A18AA6EEE00D48C09D6A5037A757E888 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:59,951 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_STARTED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, taskId=task_1463536079754_0001_1_00_000000, scheduledTime=1463536079950, launchTime=1463536079950
2016-05-18 01:47:59,953 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EB455D08F78D45C1938C862CB0322F2B in vertex: A18AA6EEE00D48C09D6A5037A757E888 numBipartiteSourceTasks: 1
2016-05-18 01:47:59,953 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:47:59,954 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:59,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536079754_0001_00_000001, launchTime=1463536079956
2016-05-18 01:47:59,957 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079754_0001_00_000001
2016-05-18 01:47:59,958 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079754_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,958 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536079754_0001_1_00_000000_0] started. Is using containerId: [container_1463536079754_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:59,958 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0, startTime=1463536079958, containerId=container_1463536079754_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:59 Starting to run new task attempt: attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,959 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536079754_0001, with dag id: dag_1463536079754_0001_1
2016-05-18 01:47:59,959 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,959 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:59,959 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:59,960 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true, VertexName: EB455D08F78D45C1938C862CB0322F2B, VertexParallelism: 1, TaskAttemptID:attempt_1463536079754_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AA9D1AAC54FC40ABB121876964D8D2E7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A18AA6EEE00D48C09D6A5037A757E888, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:59,961 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:59,961 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,962 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536079754_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:59,962 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536079754_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:59,964 INFO input.MRInput (MRInput.java:initialize(440)) - AA9D1AAC54FC40ABB121876964D8D2E7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:59,964 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: AA9D1AAC54FC40ABB121876964D8D2E7
2016-05-18 01:47:59,965 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:59,971 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EB455D08F78D45C1938C862CB0322F2B, ordinal: 0
2016-05-18 01:47:59,971 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EB455D08F78D45C1938C862CB0322F2B, mem on start (mb), free: 217, total: 348, max: 672
2016-05-18 01:47:59,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:59,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:59,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:59,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:59,971 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AA9D1AAC54FC40ABB121876964D8D2E7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A18AA6EEE00D48C09D6A5037A757E888:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:59,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:59,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:59,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:59,972 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AA9D1AAC54FC40ABB121876964D8D2E7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A18AA6EEE00D48C09D6A5037A757E888:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AA9D1AAC54FC40ABB121876964D8D2E7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AA9D1AAC54FC40ABB121876964D8D2E7
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:59,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:59,974 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,975 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:47:59,975 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - AA9D1AAC54FC40ABB121876964D8D2E7 initialized RecordReader from event
2016-05-18 01:47:59,977 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,977 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: AA9D1AAC54FC40ABB121876964D8D2E7
2016-05-18 01:47:59,977 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: CA26B6CE102C4079B15E58F612EF5275
2016-05-18 01:47:59,977 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] CA26B6CE102C4079B15E58F612EF5275
2016-05-18 01:47:59,979 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A18AA6EEE00D48C09D6A5037A757E888 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@68911657, comparator=cascading.tuple.hadoop.util.ReverseTupleComparator@d51df63, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:59,979 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A18AA6EEE00D48C09D6A5037A757E888: , UsingHashComparator=false
2016-05-18 01:47:59,991 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:59,991 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:59,991 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A18AA6EEE00D48C09D6A5037A757E888: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:59,992 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A18AA6EEE00D48C09D6A5037A757E888: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:59,992 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:47:59,992 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EB455D08F78D45C1938C862CB0322F2B, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:59,997 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:47:59,998 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EB455D08F78D45C1938C862CB0322F2B, mem on close (mb), free: 115, total: 348, max: 672
2016-05-18 01:47:59,998 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A18AA6EEE00D48C09D6A5037A757E888: Starting flush of map output
2016-05-18 01:47:59,998 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A18AA6EEE00D48C09D6A5037A757E888: Span0.length = 37, perItem = 17
2016-05-18 01:47:59,998 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A18AA6EEE00D48C09D6A5037A757E888: done sorting span=0, length=37, time=0
2016-05-18 01:47:59,998 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A18AA6EEE00D48C09D6A5037A757E888: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:48:00,000 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A18AA6EEE00D48C09D6A5037A757E888: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-local-dir/output/attempt_1463536079754_0001_1_00_000000_0_10107_0/file.out
2016-05-18 01:48:00,004 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:00 Completed running task attempt: attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:48:00,005 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:00,006 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536079754_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:00,009 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536079754_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536079992, Process_Duration=5, Process_End_Time=1463536079997, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:00,010 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536079754_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,010 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,011 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:00,011 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,011 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, taskAttemptId=attempt_1463536079754_0001_1_00_000000_0, creationTime=1463536079953, allocationTime=1463536079955, startTime=1463536079958, finishTime=1463536080011, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:00,012 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536079754_0001_1_00_000000_0
2016-05-18 01:48:00,012 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079754_0001_00_000001
2016-05-18 01:48:00,013 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_FINISHED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, taskId=task_1463536079754_0001_1_00_000000, startTime=1463536079958, finishTime=1463536080012, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463536079754_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536079992, Process_Duration=5, Process_End_Time=1463536079997, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:00,013 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:00,013 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:48:00,014 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_FINISHED]: vertexName=EB455D08F78D45C1938C862CB0322F2B, vertexId=vertex_1463536079754_0001_1_00, initRequestedTime=1463536079931, initedTime=1463536079946, startRequestedTime=1463536079933, startedTime=1463536079947, finishTime=1463536080014, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536079992, Process_Duration=5, Process_End_Time=1463536079997, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536079958, firstTasksToStart=[ task_1463536079754_0001_1_00_000000 ], lastTaskFinishTime=1463536080011, lastTasksToFinish=[ task_1463536079754_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536079754_0001_1_00_000000 ], longestDurationTasks=[ task_1463536079754_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:00,015 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:00,015 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536079754_0001_00_000001, stoppedTime=1463536080015, exitStatus=0
2016-05-18 01:48:00,015 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] attempt: attempt_1463536079754_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:00,015 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:00,015 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,015 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A18AA6EEE00D48C09D6A5037A757E888
2016-05-18 01:48:00,016 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EB455D08F78D45C1938C862CB0322F2B, destinationVertexName=A18AA6EEE00D48C09D6A5037A757E888. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:00,016 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:48:00,016 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536079754_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:00,016 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_STARTED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, taskId=task_1463536079754_0001_1_01_000000, scheduledTime=1463536080016, launchTime=1463536080016
2016-05-18 01:48:00,016 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:48:00,018 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536079754_0001_00_000002, launchTime=1463536080018
2016-05-18 01:48:00,019 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079754_0001_00_000002
2016-05-18 01:48:00,020 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079754_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,020 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536079754_0001_1_01_000000_0] started. Is using containerId: [container_1463536079754_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:00 Starting to run new task attempt: attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,020 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,020 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,021 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:00,021 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true, VertexName: A18AA6EEE00D48C09D6A5037A757E888, VertexParallelism: 1, TaskAttemptID:attempt_1463536079754_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EB455D08F78D45C1938C862CB0322F2B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=41D9FE0855424CED8E46CD99EBD51409, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:00,021 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:00,021 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0, startTime=1463536080020, containerId=container_1463536079754_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:00,021 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,023 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536079754_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:48:00,023 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536079754_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:00,023 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536079754_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:00,023 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079754_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:00,023 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536079754_0001_00_000001, Exiting
2016-05-18 01:48:00,023 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536079754_0001_00_000001
2016-05-18 01:48:00,024 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536079754_0001_00_000001
2016-05-18 01:48:00,025 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:00,026 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536079754_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:00,027 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536079754_0001_00_000001 completed successfully
2016-05-18 01:48:00,029 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:00,035 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A18AA6EEE00D48C09D6A5037A757E888, ordinal: 1
2016-05-18 01:48:00,035 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A18AA6EEE00D48C09D6A5037A757E888, mem on start (mb), free: 110, total: 348, max: 672
2016-05-18 01:48:00,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:00,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:00,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:00,041 INFO output.MROutput (MROutput.java:initialize(421)) - 41D9FE0855424CED8E46CD99EBD51409: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:00,041 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:00,041 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EB455D08F78D45C1938C862CB0322F2B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [41D9FE0855424CED8E46CD99EBD51409:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:00,041 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:00,041 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:00,041 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:00,041 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EB455D08F78D45C1938C862CB0322F2B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [41D9FE0855424CED8E46CD99EBD51409:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:00,041 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:00,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EB455D08F78D45C1938C862CB0322F2B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:00,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:00,042 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:00,042 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EB455D08F78D45C1938C862CB0322F2B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:00,042 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EB455D08F78D45C1938C862CB0322F2B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:00,043 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EB455D08F78D45C1938C862CB0322F2B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:00,046 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EB455D08F78D45C1938C862CB0322F2B
2016-05-18 01:48:00,046 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:00,047 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,051 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:00,051 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EB455D08F78D45C1938C862CB0322F2B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:00,051 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: CA26B6CE102C4079B15E58F612EF5275
2016-05-18 01:48:00,051 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true"], id: 41D9FE0855424CED8E46CD99EBD51409
2016-05-18 01:48:00,052 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true"]
2016-05-18 01:48:00,052 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] CA26B6CE102C4079B15E58F612EF5275, for 1 inputs
2016-05-18 01:48:00,055 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536079754_0001_1_00_000000_0_10107} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536080055, TimeTaken=2, Rate=0.28 MB/s
2016-05-18 01:48:00,055 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EB455D08F78D45C1938C862CB0322F2B
2016-05-18 01:48:00,056 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:00,056 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EB455D08F78D45C1938C862CB0322F2B, wasInterrupted=false
2016-05-18 01:48:00,056 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:00,058 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:00,059 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001_wd/localmode-local-dir/output/attempt_1463536079754_0001_1_00_000000_0_10107/file.out, len=593, isLocal=true
2016-05-18 01:48:00,059 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - EB455D08F78D45C1938C862CB0322F2B: Already shutdown. Ignoring fetch complete
2016-05-18 01:48:00,059 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:48:00,059 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:00,059 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:00,059 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:48:00,059 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EB455D08F78D45C1938C862CB0322F2B
2016-05-18 01:48:00,060 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EB455D08F78D45C1938C862CB0322F2B: Shuffle Runner thread complete
2016-05-18 01:48:00,060 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A18AA6EEE00D48C09D6A5037A757E888, all 1 inputs ready in: 00:00:00.008
2016-05-18 01:48:00,061 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EB455D08F78D45C1938C862CB0322F2B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseTupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:00,067 INFO output.MROutput (MROutput.java:close(526)) - 41D9FE0855424CED8E46CD99EBD51409 closed
2016-05-18 01:48:00,067 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536079754_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:00,069 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,069 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A18AA6EEE00D48C09D6A5037A757E888, mem on close (mb), free: 105, total: 348, max: 672
2016-05-18 01:48:00,070 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EB455D08F78D45C1938C862CB0322F2B
2016-05-18 01:48:00,070 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EB455D08F78D45C1938C862CB0322F2B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:00,071 INFO output.MROutput (MROutput.java:close(526)) - 41D9FE0855424CED8E46CD99EBD51409 closed
2016-05-18 01:48:00,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:00 Completed running task attempt: attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,074 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:00,074 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536079754_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:00,075 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536079754_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536080060, Process_Duration=9, Process_End_Time=1463536080069, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:00,077 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536079754_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, taskAttemptId=attempt_1463536079754_0001_1_01_000000_0, creationTime=1463536080016, allocationTime=1463536080017, startTime=1463536080020, finishTime=1463536080077, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:00,077 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,077 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:00,077 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,077 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536079754_0001_00_000002
2016-05-18 01:48:00,077 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536079754_0001_1_01_000000_0
2016-05-18 01:48:00,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:TASK_FINISHED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, taskId=task_1463536079754_0001_1_01_000000, startTime=1463536080020, finishTime=1463536080078, timeTaken=58, status=SUCCEEDED, successfulAttemptID=attempt_1463536079754_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536080060, Process_Duration=9, Process_End_Time=1463536080069, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:00,078 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:00,079 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888]
2016-05-18 01:48:00,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:VERTEX_FINISHED]: vertexName=A18AA6EEE00D48C09D6A5037A757E888, vertexId=vertex_1463536079754_0001_1_01, initRequestedTime=1463536079934, initedTime=1463536079939, startRequestedTime=1463536079948, startedTime=1463536079948, finishTime=1463536080079, timeTaken=131, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536080060, Process_Duration=9, Process_End_Time=1463536080069, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536080020, firstTasksToStart=[ task_1463536079754_0001_1_01_000000 ], lastTaskFinishTime=1463536080077, lastTasksToFinish=[ task_1463536079754_0001_1_01_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536079754_0001_1_01_000000 ], longestDurationTasks=[ task_1463536079754_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:00,080 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:00,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536079754_0001_00_000002, stoppedTime=1463536080080, exitStatus=0
2016-05-18 01:48:00,080 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:00,080 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,080 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536079754_0001_1_00 [EB455D08F78D45C1938C862CB0322F2B]
2016-05-18 01:48:00,081 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536079754_0001_1
2016-05-18 01:48:00,081 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536079754_0001_1
2016-05-18 01:48:00,081 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079754_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:00,081 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 41D9FE0855424CED8E46CD99EBD51409 for vertex: vertex_1463536079754_0001_1_01 [A18AA6EEE00D48C09D6A5037A757E888], outputName: 41D9FE0855424CED8E46CD99EBD51409
2016-05-18 01:48:00,082 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536079754_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:00,088 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536079754_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:00,088 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536079754_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:00,088 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536079754_0001_00_000002, Exiting
2016-05-18 01:48:00,089 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536079754_0001_00_000002
2016-05-18 01:48:00,089 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536079754_0001_00_000002
2016-05-18 01:48:00,089 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536079754_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:00,089 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536079754_0001_00_000002 completed successfully
2016-05-18 01:48:00,093 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:41D9FE0855424CED8E46CD99EBD51409 of vertex/vertexGroup:A18AA6EEE00D48C09D6A5037A757E888 isVertexGroupOutput:false
2016-05-18 01:48:00,094 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,096 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536079754_0001_1, queueSize=0
2016-05-18 01:48:00,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536079754_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536079754_0001_1, startTime=1463536079931, finishTime=1463536080094, timeTaken=163, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=350, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072160052, Process_Duration=14, Process_End_Time=2927072160066, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:00,096 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536079754_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:00,096 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536079754_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:00 Completed Dag: dag_1463536079754_0001_1
2016-05-18 01:48:00,097 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536079754_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:00,097 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536079754_0001_1_post
2016-05-18 01:48:00,097 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,097 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:00,097 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:00,097 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true, with id=dag_1463536079754_0001_1
2016-05-18 01:48:00,098 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5] comprehensivesort/(1/1) ...sive/false_null_true_true, with id=dag_1463536079754_0001_1
2016-05-18 01:48:00,109 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/false_null_true_true, applicationId=application_1463536079754_0001
2016-05-18 01:48:00,111 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:00,111 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:00,111 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:00,112 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:00,112 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:00,112 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:00,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:00,114 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:00,114 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:00,114 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:00,115 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:00,115 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:00,115 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:00,116 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:00,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/10084775424F4704BECA8B06D5B1F024/EC94095AF5F644208CBD34DCA9825BC5/.tez/application_1463536079754_0001
2016-05-18 01:48:00,120 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true/_temporary
2016-05-18 01:48:00,121 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.386
2016-05-18 01:48:00,147 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:00,147 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:00,216 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:00,216 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:00,253 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 01:48:00,264 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.023
2016-05-18 01:48:00,265 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:00,265 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:00,266 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:00,280 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:00,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true"]
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:00,335 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:00,336 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...nsive/true_null_true_true
2016-05-18 01:48:00,336 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:00,351 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:00,355 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:00,355 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:00,355 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:00,365 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001 doesn't exist and is created
2016-05-18 01:48:00,385 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:00,385 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:00,386 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd
2016-05-18 01:48:00,402 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536080355_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:00,402 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:00,402 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:00,403 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:00,404 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:00,404 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:00,404 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:00,405 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:00,406 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:00,406 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:00,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:00,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:00,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:00,407 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:00,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:00,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:00,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:00,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:00,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:00,408 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:00,410 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:00,411 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:00,411 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:00,411 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-log-dir
2016-05-18 01:48:00,412 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-log-dir/history.txt.appattempt_1463536080355_0001_000000, maxErrors=10
2016-05-18 01:48:00,412 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:00,412 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536080355_0001_000000, appSubmitTime=1463536080401, launchTime=1463536080401
2016-05-18 01:48:00,418 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:00,420 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42127
2016-05-18 01:48:00,420 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536080355
2016-05-18 01:48:00,421 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536080355_0001_000000, startTime=1463536080421
2016-05-18 01:48:00,422 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:00,486 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:00,486 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:00,486 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...nsive/true_null_true_true, applicationId=application_1463536080355_0001, dagName=[E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true
2016-05-18 01:48:00,502 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true
2016-05-18 01:48:00,502 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:00,503 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536080355_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-log-dir/dag_1463536080355_0001_1.dot
2016-05-18 01:48:00,503 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-log-dir/dag_1463536080355_0001_1-tez-dag.pb.txt
2016-05-18 01:48:00,509 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536080355_0001_1
2016-05-18 01:48:00,509 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,510 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true
2016-05-18 01:48:00 Running Dag: dag_1463536080355_0001_1
2016-05-18 01:48:00,524 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536080355_0001_1, submitTime=1463536080502
2016-05-18 01:48:00,526 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001/tez-conf.pb" } size: 40937 timestamp: 1463536080000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:00,526 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,527 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:00,527 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001/tez-conf.pb" } size: 40937 timestamp: 1463536080000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:00,528 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,528 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:00,528 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:00,529 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536080355_0001_1, initTime=1463536080529
2016-05-18 01:48:00,529 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080355_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:00,529 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:00,530 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:DAG_STARTED]: dagID=dag_1463536080355_0001_1, startTime=1463536080530
2016-05-18 01:48:00,530 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080355_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:00,530 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C07CA7472E534DFEAE859C2520760899 : {7E698DFED14549AFBCA9C76C79F226EF={InputName=7E698DFED14549AFBCA9C76C79F226EF}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:00,530 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 7E698DFED14549AFBCA9C76C79F226EF, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:00,530 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,530 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,531 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,531 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,532 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:00,532 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,534 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...nsive/true_null_true_true, applicationId=application_1463536080355_0001, dagName=[E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true
2016-05-18 01:48:00,535 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:00,536 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,536 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,536 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 7E698DFED14549AFBCA9C76C79F226EF on vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,536 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536080355_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:00,538 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188], numAdditionalOutputs=1
2016-05-18 01:48:00,538 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=5AE97322586746F292FD6E51EE515433, vertex=vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:00,539 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 7E698DFED14549AFBCA9C76C79F226EF asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:00,540 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DAFE9F96AAE946B794BF70EDD8427188:5AE97322586746F292FD6E51EE515433 using oldmapred API
2016-05-18 01:48:00,541 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=5AE97322586746F292FD6E51EE515433, vertexName=DAFE9F96AAE946B794BF70EDD8427188, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:00,542 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, vertexId=vertex_1463536080355_0001_1_01, initRequestedTime=1463536080532, initedTime=1463536080538, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:00,542 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:00,543 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:00,544 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:00,544 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:00,544 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:00,545 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 7E698DFED14549AFBCA9C76C79F226EF on vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,546 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:00,546 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {7E698DFED14549AFBCA9C76C79F226EF=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:00,547 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] parallelism set to 1
2016-05-18 01:48:00,547 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:00,548 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C07CA7472E534DFEAE859C2520760899, vertexId=vertex_1463536080355_0001_1_00, initRequestedTime=1463536080530, initedTime=1463536080547, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:00,548 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] with distanceFromRoot: 0
2016-05-18 01:48:00,548 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:00,549 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536080355_0001_1_00, startRequestedTime=1463536080532, startedTime=1463536080548
2016-05-18 01:48:00,549 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536080355_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:00,549 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:00,550 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536080355_0001_1_00 for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] numStartedSources: 1 numSources: 1
2016-05-18 01:48:00,550 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] with distanceFromRoot: 1
2016-05-18 01:48:00,550 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C07CA7472E534DFEAE859C2520760899
2016-05-18 01:48:00,551 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,551 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536080355_0001_1_01, startRequestedTime=1463536080550, startedTime=1463536080550
2016-05-18 01:48:00,551 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:00,552 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_STARTED]: vertexName=C07CA7472E534DFEAE859C2520760899, taskId=task_1463536080355_0001_1_00_000000, scheduledTime=1463536080552, launchTime=1463536080552
2016-05-18 01:48:00,552 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,554 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DAFE9F96AAE946B794BF70EDD8427188 with 0 source tasks and 1 pending tasks
2016-05-18 01:48:00,555 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C07CA7472E534DFEAE859C2520760899 in vertex: DAFE9F96AAE946B794BF70EDD8427188 numBipartiteSourceTasks: 1
2016-05-18 01:48:00,555 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:00,557 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536080355_0001_00_000001, launchTime=1463536080557
2016-05-18 01:48:00,557 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080355_0001_00_000001
2016-05-18 01:48:00,558 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536080355_0001_1_00_000000_0] started. Is using containerId: [container_1463536080355_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:00,558 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C07CA7472E534DFEAE859C2520760899, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0, startTime=1463536080558, containerId=container_1463536080355_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:00,559 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080355_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00 Starting to run new task attempt: attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,560 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,560 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,560 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:00,560 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true, VertexName: C07CA7472E534DFEAE859C2520760899, VertexParallelism: 1, TaskAttemptID:attempt_1463536080355_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=7E698DFED14549AFBCA9C76C79F226EF, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DAFE9F96AAE946B794BF70EDD8427188, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:00,560 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:00,561 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,562 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536080355_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:00,562 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536080355_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:00,563 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536080355_0001, with dag id: dag_1463536080355_0001_1
2016-05-18 01:48:00,563 INFO input.MRInput (MRInput.java:initialize(440)) - 7E698DFED14549AFBCA9C76C79F226EF using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:00,563 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 7E698DFED14549AFBCA9C76C79F226EF
2016-05-18 01:48:00,567 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:00,573 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C07CA7472E534DFEAE859C2520760899, ordinal: 0
2016-05-18 01:48:00,573 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C07CA7472E534DFEAE859C2520760899, mem on start (mb), free: 186, total: 347, max: 672
2016-05-18 01:48:00,573 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:00,574 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:00,574 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:00,574 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:00,574 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[7E698DFED14549AFBCA9C76C79F226EF:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DAFE9F96AAE946B794BF70EDD8427188:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:00,575 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:00,575 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:00,575 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:00,575 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[7E698DFED14549AFBCA9C76C79F226EF:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DAFE9F96AAE946B794BF70EDD8427188:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:00,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:00,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 7E698DFED14549AFBCA9C76C79F226EF
2016-05-18 01:48:00,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 7E698DFED14549AFBCA9C76C79F226EF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:00,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:00,577 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:00,577 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:00,577 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,579 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:00,579 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 7E698DFED14549AFBCA9C76C79F226EF initialized RecordReader from event
2016-05-18 01:48:00,580 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:00,581 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 7E698DFED14549AFBCA9C76C79F226EF
2016-05-18 01:48:00,581 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: A8324BB056D845D8B0806CED557D42AA
2016-05-18 01:48:00,581 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A8324BB056D845D8B0806CED557D42AA
2016-05-18 01:48:00,583 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DAFE9F96AAE946B794BF70EDD8427188 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@20186681, comparator=cascading.tuple.hadoop.util.ReverseTupleComparator@21f1af9d, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:00,584 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DAFE9F96AAE946B794BF70EDD8427188: , UsingHashComparator=false
2016-05-18 01:48:00,614 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:00,614 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:00,615 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DAFE9F96AAE946B794BF70EDD8427188: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:00,615 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DAFE9F96AAE946B794BF70EDD8427188: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:00,615 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:00,616 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C07CA7472E534DFEAE859C2520760899, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:00,620 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,620 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C07CA7472E534DFEAE859C2520760899, mem on close (mb), free: 190, total: 341, max: 672
2016-05-18 01:48:00,621 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DAFE9F96AAE946B794BF70EDD8427188: Starting flush of map output
2016-05-18 01:48:00,621 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DAFE9F96AAE946B794BF70EDD8427188: Span0.length = 37, perItem = 17
2016-05-18 01:48:00,622 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DAFE9F96AAE946B794BF70EDD8427188: done sorting span=0, length=37, time=1
2016-05-18 01:48:00,622 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DAFE9F96AAE946B794BF70EDD8427188: Heap = SpanIterator<0:36> (span=Span[16777216,629]),
2016-05-18 01:48:00,623 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DAFE9F96AAE946B794BF70EDD8427188: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-local-dir/output/attempt_1463536080355_0001_1_00_000000_0_10113_0/file.out
2016-05-18 01:48:00,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:00 Completed running task attempt: attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,627 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:00,627 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536080355_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:00,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536080355_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536080616, Process_Duration=4, Process_End_Time=1463536080620, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:00,629 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536080355_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,629 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,629 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:00,629 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C07CA7472E534DFEAE859C2520760899, taskAttemptId=attempt_1463536080355_0001_1_00_000000_0, creationTime=1463536080552, allocationTime=1463536080556, startTime=1463536080558, finishTime=1463536080630, timeTaken=72, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:00,630 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536080355_0001_1_00_000000_0
2016-05-18 01:48:00,630 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080355_0001_00_000001
2016-05-18 01:48:00,631 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_FINISHED]: vertexName=C07CA7472E534DFEAE859C2520760899, taskId=task_1463536080355_0001_1_00_000000, startTime=1463536080558, finishTime=1463536080630, timeTaken=72, status=SUCCEEDED, successfulAttemptID=attempt_1463536080355_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536080616, Process_Duration=4, Process_End_Time=1463536080620, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:00,631 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:00,632 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_FINISHED]: vertexName=C07CA7472E534DFEAE859C2520760899, vertexId=vertex_1463536080355_0001_1_00, initRequestedTime=1463536080530, initedTime=1463536080547, startRequestedTime=1463536080532, startedTime=1463536080548, finishTime=1463536080632, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536080616, Process_Duration=4, Process_End_Time=1463536080620, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536080558, firstTasksToStart=[ task_1463536080355_0001_1_00_000000 ], lastTaskFinishTime=1463536080630, lastTasksToFinish=[ task_1463536080355_0001_1_00_000000 ], minTaskDuration=72, maxTaskDuration=72, avgTaskDuration=72.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536080355_0001_1_00_000000 ], longestDurationTasks=[ task_1463536080355_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:00,633 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:00,633 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] attempt: attempt_1463536080355_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:00,633 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DAFE9F96AAE946B794BF70EDD8427188
2016-05-18 01:48:00,633 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536080355_0001_00_000001, stoppedTime=1463536080633, exitStatus=0
2016-05-18 01:48:00,634 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C07CA7472E534DFEAE859C2520760899, destinationVertexName=DAFE9F96AAE946B794BF70EDD8427188. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:00,634 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,634 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:00,634 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,634 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536080355_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:00,634 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_STARTED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, taskId=task_1463536080355_0001_1_01_000000, scheduledTime=1463536080634, launchTime=1463536080634
2016-05-18 01:48:00,635 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,637 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536080355_0001_00_000002, launchTime=1463536080636
2016-05-18 01:48:00,637 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080355_0001_00_000002
2016-05-18 01:48:00,638 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536080355_0001_1_01_000000_0] started. Is using containerId: [container_1463536080355_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:00,638 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0, startTime=1463536080638, containerId=container_1463536080355_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:00,638 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080355_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00 Starting to run new task attempt: attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,639 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,639 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,640 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:00,640 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true, VertexName: DAFE9F96AAE946B794BF70EDD8427188, VertexParallelism: 1, TaskAttemptID:attempt_1463536080355_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C07CA7472E534DFEAE859C2520760899, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=5AE97322586746F292FD6E51EE515433, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:00,640 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:00,641 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,641 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536080355_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:00,641 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080355_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:00,641 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536080355_0001_00_000001, Exiting
2016-05-18 01:48:00,641 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536080355_0001_00_000001
2016-05-18 01:48:00,641 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536080355_0001_00_000001
2016-05-18 01:48:00,642 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536080355_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,642 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536080355_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:00,644 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:00,645 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:00,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536080355_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:00,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536080355_0001_00_000001 completed successfully
2016-05-18 01:48:00,651 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DAFE9F96AAE946B794BF70EDD8427188, ordinal: 1
2016-05-18 01:48:00,651 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DAFE9F96AAE946B794BF70EDD8427188, mem on start (mb), free: 183, total: 341, max: 672
2016-05-18 01:48:00,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:00,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:00,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:00,660 INFO output.MROutput (MROutput.java:initialize(421)) - 5AE97322586746F292FD6E51EE515433: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:00,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:00,660 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C07CA7472E534DFEAE859C2520760899:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [5AE97322586746F292FD6E51EE515433:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:00,660 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:00,660 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:00,660 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:00,661 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C07CA7472E534DFEAE859C2520760899:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [5AE97322586746F292FD6E51EE515433:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:00,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:00,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C07CA7472E534DFEAE859C2520760899 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:00,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:00,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:00,662 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C07CA7472E534DFEAE859C2520760899: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:00,662 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C07CA7472E534DFEAE859C2520760899: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:00,663 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C07CA7472E534DFEAE859C2520760899 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:00,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C07CA7472E534DFEAE859C2520760899
2016-05-18 01:48:00,666 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:00,666 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,669 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:00,670 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C07CA7472E534DFEAE859C2520760899: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:00,670 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: A8324BB056D845D8B0806CED557D42AA
2016-05-18 01:48:00,671 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true"], id: 5AE97322586746F292FD6E51EE515433
2016-05-18 01:48:00,671 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true"]
2016-05-18 01:48:00,672 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A8324BB056D845D8B0806CED557D42AA, for 1 inputs
2016-05-18 01:48:00,673 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536080355_0001_1_00_000000_0_10113} to DISK_DIRECT, csize=593, dsize=589, EndTime=1463536080673, TimeTaken=1, Rate=0.57 MB/s
2016-05-18 01:48:00,673 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C07CA7472E534DFEAE859C2520760899
2016-05-18 01:48:00,673 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:00,675 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C07CA7472E534DFEAE859C2520760899, wasInterrupted=false
2016-05-18 01:48:00,676 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:00,676 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:00,677 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001_wd/localmode-local-dir/output/attempt_1463536080355_0001_1_00_000000_0_10113/file.out, len=593, isLocal=true
2016-05-18 01:48:00,677 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 593 bytes from disk
2016-05-18 01:48:00,677 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:00,677 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:00,678 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 580 bytes
2016-05-18 01:48:00,678 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C07CA7472E534DFEAE859C2520760899
2016-05-18 01:48:00,678 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C07CA7472E534DFEAE859C2520760899: Shuffle Runner thread complete
2016-05-18 01:48:00,678 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DAFE9F96AAE946B794BF70EDD8427188, all 1 inputs ready in: 00:00:00.006
2016-05-18 01:48:00,679 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C07CA7472E534DFEAE859C2520760899: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseTupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:00,684 INFO output.MROutput (MROutput.java:close(526)) - 5AE97322586746F292FD6E51EE515433 closed
2016-05-18 01:48:00,685 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536080355_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:00,685 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,685 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DAFE9F96AAE946B794BF70EDD8427188, mem on close (mb), free: 179, total: 341, max: 672
2016-05-18 01:48:00,686 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C07CA7472E534DFEAE859C2520760899
2016-05-18 01:48:00,686 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C07CA7472E534DFEAE859C2520760899: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:00,686 INFO output.MROutput (MROutput.java:close(526)) - 5AE97322586746F292FD6E51EE515433 closed
2016-05-18 01:48:00,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:00 Completed running task attempt: attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,687 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:00,687 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536080355_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:00,687 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536080355_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536080678, Process_Duration=7, Process_End_Time=1463536080685, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:00,688 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536080355_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, taskAttemptId=attempt_1463536080355_0001_1_01_000000_0, creationTime=1463536080635, allocationTime=1463536080636, startTime=1463536080638, finishTime=1463536080688, timeTaken=50, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:00,689 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536080355_0001_1_01_000000_0
2016-05-18 01:48:00,689 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:TASK_FINISHED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, taskId=task_1463536080355_0001_1_01_000000, startTime=1463536080638, finishTime=1463536080689, timeTaken=51, status=SUCCEEDED, successfulAttemptID=attempt_1463536080355_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536080678, Process_Duration=7, Process_End_Time=1463536080685, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:00,689 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:00,690 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188]
2016-05-18 01:48:00,688 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:00,690 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:00,690 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,690 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:VERTEX_FINISHED]: vertexName=DAFE9F96AAE946B794BF70EDD8427188, vertexId=vertex_1463536080355_0001_1_01, initRequestedTime=1463536080532, initedTime=1463536080538, startRequestedTime=1463536080550, startedTime=1463536080550, finishTime=1463536080690, timeTaken=140, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536080678, Process_Duration=7, Process_End_Time=1463536080685, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536080638, firstTasksToStart=[ task_1463536080355_0001_1_01_000000 ], lastTaskFinishTime=1463536080688, lastTasksToFinish=[ task_1463536080355_0001_1_01_000000 ], minTaskDuration=50, maxTaskDuration=50, avgTaskDuration=50.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536080355_0001_1_01_000000 ], longestDurationTasks=[ task_1463536080355_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:00,691 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080355_0001_00_000002
2016-05-18 01:48:00,691 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:00,691 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536080355_0001_00_000002, stoppedTime=1463536080691, exitStatus=0
2016-05-18 01:48:00,691 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:00,691 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,691 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536080355_0001_1_00 [C07CA7472E534DFEAE859C2520760899]
2016-05-18 01:48:00,691 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536080355_0001_1
2016-05-18 01:48:00,692 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536080355_0001_1
2016-05-18 01:48:00,692 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080355_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:00,692 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 5AE97322586746F292FD6E51EE515433 for vertex: vertex_1463536080355_0001_1_01 [DAFE9F96AAE946B794BF70EDD8427188], outputName: 5AE97322586746F292FD6E51EE515433
2016-05-18 01:48:00,692 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536080355_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:00,701 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:5AE97322586746F292FD6E51EE515433 of vertex/vertexGroup:DAFE9F96AAE946B794BF70EDD8427188 isVertexGroupOutput:false
2016-05-18 01:48:00,701 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536080355_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:00,701 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080355_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:00,701 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536080355_0001_00_000002, Exiting
2016-05-18 01:48:00,701 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:00,702 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536080355_0001_00_000002
2016-05-18 01:48:00,702 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536080355_0001_00_000002
2016-05-18 01:48:00,702 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536080355_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:00,702 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536080355_0001_00_000002 completed successfully
2016-05-18 01:48:00,704 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536080355_0001_1, queueSize=0
2016-05-18 01:48:00,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080355_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536080355_0001_1, startTime=1463536080529, finishTime=1463536080702, timeTaken=173, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=330, AM_GC_TIME_MILLIS=17, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=629, OUTPUT_BYTES_WITH_OVERHEAD=589, OUTPUT_BYTES_PHYSICAL=593, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=593, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=593, SHUFFLE_BYTES_DECOMPRESSED=589, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=593, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072161294, Process_Duration=11, Process_End_Time=2927072161305, Read_Duration=1, Tuples_Read=74, Tuples_Written=74, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:00,704 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536080355_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:00,704 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080355_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:00 Completed Dag: dag_1463536080355_0001_1
2016-05-18 01:48:00,705 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536080355_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:00,705 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536080355_0001_1_post
2016-05-18 01:48:00,706 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:00,706 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:00,706 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:00,706 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true, with id=dag_1463536080355_0001_1
2016-05-18 01:48:00,707 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197] comprehensivesort/(1/1) ...nsive/true_null_true_true, with id=dag_1463536080355_0001_1
2016-05-18 01:48:00,713 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...nsive/true_null_true_true, applicationId=application_1463536080355_0001
2016-05-18 01:48:00,715 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:00,715 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:00,715 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:00,715 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:00,715 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:00,716 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:00,716 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:00,716 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:00,716 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:00,717 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:00,717 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:00,717 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:00,717 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:00,718 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:00,722 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/E39E2AB1BCFF4E8B94ABA83D9E17DAFD/CC687B8D5D264140ACF1CEDC0DED2197/.tez/application_1463536080355_0001
2016-05-18 01:48:00,723 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true/_temporary
2016-05-18 01:48:00,723 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.388
2016-05-18 01:48:00,762 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:00,762 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:00,797 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:00,798 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:00,836 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 01:48:00,844 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.019
2016-05-18 01:48:00,844 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:00,844 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:00,845 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:00,861 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:00,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true
2016-05-18 01:48:00,916 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:00,917 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:00,918 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true"]
2016-05-18 01:48:00,918 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:00,918 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:00,918 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:00,919 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ve/false_false_false_true
2016-05-18 01:48:00,919 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:00,932 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:00,936 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:00,937 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:00,937 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:00,946 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001 doesn't exist and is created
2016-05-18 01:48:00,966 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:00,966 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:00,967 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd
2016-05-18 01:48:00,982 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536080937_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:00,982 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:00,982 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:00,984 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:00,984 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:00,984 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:00,984 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:00,984 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:00,986 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:00,986 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:00,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:00,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:00,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:00,987 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:00,987 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:00,989 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:00,989 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:00,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:00,990 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-log-dir
2016-05-18 01:48:00,990 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-log-dir/history.txt.appattempt_1463536080937_0001_000000, maxErrors=10
2016-05-18 01:48:00,990 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:00,990 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536080937_0001_000000, appSubmitTime=1463536080982, launchTime=1463536080982
2016-05-18 01:48:00,992 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:01,021 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60870
2016-05-18 01:48:01,022 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536080937
2016-05-18 01:48:01,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536080937_0001_000000, startTime=1463536081023
2016-05-18 01:48:01,024 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:01,067 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:01,067 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:01,067 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ve/false_false_false_true, applicationId=application_1463536080937_0001, dagName=[4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true
2016-05-18 01:48:01,085 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true
2016-05-18 01:48:01,085 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:01,086 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536080937_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-log-dir/dag_1463536080937_0001_1.dot
2016-05-18 01:48:01,087 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-log-dir/dag_1463536080937_0001_1-tez-dag.pb.txt
2016-05-18 01:48:01,093 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536080937_0001_1
2016-05-18 01:48:01,093 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,093 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true
2016-05-18 01:48:01 Running Dag: dag_1463536080937_0001_1
2016-05-18 01:48:01,108 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536080937_0001_1, submitTime=1463536081085
2016-05-18 01:48:01,109 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001/tez-conf.pb" } size: 40937 timestamp: 1463536080000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:01,110 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,110 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:01,111 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001/tez-conf.pb" } size: 40937 timestamp: 1463536080000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:01,112 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,112 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:01,112 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:01,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536080937_0001_1, initTime=1463536081113
2016-05-18 01:48:01,114 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080937_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:01,114 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:01,115 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:DAG_STARTED]: dagID=dag_1463536080937_0001_1, startTime=1463536081115
2016-05-18 01:48:01,115 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080937_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:01,115 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CC86AB31E5C74E749A360A351BB4001A : {DF8F4F43ED23410AB2FD93ED9146713E={InputName=DF8F4F43ED23410AB2FD93ED9146713E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:01,115 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: DF8F4F43ED23410AB2FD93ED9146713E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:01,115 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,116 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,116 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,116 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ve/false_false_false_true, applicationId=application_1463536080937_0001, dagName=[4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true
2016-05-18 01:48:01,116 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,116 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:01,117 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,118 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: DF8F4F43ED23410AB2FD93ED9146713E on vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,120 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:01,120 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,120 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,121 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input DF8F4F43ED23410AB2FD93ED9146713E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:01,121 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536080937_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:01,122 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A], numAdditionalOutputs=1
2016-05-18 01:48:01,122 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=540B8F8DD8A644BEB0623AAD940165E4, vertex=vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:01,123 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:01,123 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:01,123 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:01,124 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:01,124 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A47CB8FE43B24AD98EF3BA692C8E067A:540B8F8DD8A644BEB0623AAD940165E4 using oldmapred API
2016-05-18 01:48:01,124 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: DF8F4F43ED23410AB2FD93ED9146713E on vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,125 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=540B8F8DD8A644BEB0623AAD940165E4, vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:01,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, vertexId=vertex_1463536080937_0001_1_01, initRequestedTime=1463536081117, initedTime=1463536081122, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:01,126 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:01,129 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:01,129 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {DF8F4F43ED23410AB2FD93ED9146713E=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:01,129 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] parallelism set to 1
2016-05-18 01:48:01,129 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:01,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, vertexId=vertex_1463536080937_0001_1_00, initRequestedTime=1463536081115, initedTime=1463536081130, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:01,130 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] with distanceFromRoot: 0
2016-05-18 01:48:01,130 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:01,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536080937_0001_1_00, startRequestedTime=1463536081116, startedTime=1463536081130
2016-05-18 01:48:01,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536080937_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:01,130 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:01,130 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536080937_0001_1_00 for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] numStartedSources: 1 numSources: 1
2016-05-18 01:48:01,130 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] with distanceFromRoot: 1
2016-05-18 01:48:01,131 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536080937_0001_1_01, startRequestedTime=1463536081130, startedTime=1463536081130
2016-05-18 01:48:01,131 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:01,134 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A47CB8FE43B24AD98EF3BA692C8E067A with 0 source tasks and 1 pending tasks
2016-05-18 01:48:01,134 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CC86AB31E5C74E749A360A351BB4001A
2016-05-18 01:48:01,134 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CC86AB31E5C74E749A360A351BB4001A in vertex: A47CB8FE43B24AD98EF3BA692C8E067A numBipartiteSourceTasks: 1
2016-05-18 01:48:01,135 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_STARTED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, taskId=task_1463536080937_0001_1_00_000000, scheduledTime=1463536081135, launchTime=1463536081135
2016-05-18 01:48:01,135 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,136 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:01,137 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536080937_0001_00_000001, launchTime=1463536081137
2016-05-18 01:48:01,138 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080937_0001_00_000001
2016-05-18 01:48:01,140 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080937_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,140 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536080937_0001_1_00_000000_0] started. Is using containerId: [container_1463536080937_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:01,140 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0, startTime=1463536081140, containerId=container_1463536080937_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:01 Starting to run new task attempt: attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,142 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,142 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,142 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:01,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true, VertexName: CC86AB31E5C74E749A360A351BB4001A, VertexParallelism: 1, TaskAttemptID:attempt_1463536080937_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF8F4F43ED23410AB2FD93ED9146713E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A47CB8FE43B24AD98EF3BA692C8E067A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:01,143 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:01,143 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,144 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536080937_0001, with dag id: dag_1463536080937_0001_1
2016-05-18 01:48:01,146 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536080937_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:01,147 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536080937_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:01,152 INFO input.MRInput (MRInput.java:initialize(440)) - DF8F4F43ED23410AB2FD93ED9146713E using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:01,152 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: DF8F4F43ED23410AB2FD93ED9146713E
2016-05-18 01:48:01,154 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:01,158 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CC86AB31E5C74E749A360A351BB4001A, ordinal: 0
2016-05-18 01:48:01,158 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC86AB31E5C74E749A360A351BB4001A, mem on start (mb), free: 260, total: 338, max: 672
2016-05-18 01:48:01,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:01,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:01,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:01,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:01,159 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DF8F4F43ED23410AB2FD93ED9146713E:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A47CB8FE43B24AD98EF3BA692C8E067A:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:01,159 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:01,159 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:01,160 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:01,160 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DF8F4F43ED23410AB2FD93ED9146713E:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A47CB8FE43B24AD98EF3BA692C8E067A:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DF8F4F43ED23410AB2FD93ED9146713E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DF8F4F43ED23410AB2FD93ED9146713E
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:01,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:01,161 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,163 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:01,163 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - DF8F4F43ED23410AB2FD93ED9146713E initialized RecordReader from event
2016-05-18 01:48:01,164 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:01,166 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: DF8F4F43ED23410AB2FD93ED9146713E
2016-05-18 01:48:01,167 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 31B6AC1BEDDD4CB48951155E36ECADE4
2016-05-18 01:48:01,167 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 31B6AC1BEDDD4CB48951155E36ECADE4
2016-05-18 01:48:01,169 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A47CB8FE43B24AD98EF3BA692C8E067A using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@7235ee41, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@32ad27fb, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:01,169 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A47CB8FE43B24AD98EF3BA692C8E067A: , UsingHashComparator=false
2016-05-18 01:48:01,183 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:01,183 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:01,183 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A47CB8FE43B24AD98EF3BA692C8E067A: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:01,183 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A47CB8FE43B24AD98EF3BA692C8E067A: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:01,184 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:01,184 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CC86AB31E5C74E749A360A351BB4001A, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:01,188 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,189 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC86AB31E5C74E749A360A351BB4001A, mem on close (mb), free: 158, total: 338, max: 672
2016-05-18 01:48:01,189 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A47CB8FE43B24AD98EF3BA692C8E067A: Starting flush of map output
2016-05-18 01:48:01,189 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A47CB8FE43B24AD98EF3BA692C8E067A: Span0.length = 37, perItem = 24
2016-05-18 01:48:01,190 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A47CB8FE43B24AD98EF3BA692C8E067A: done sorting span=0, length=37, time=1
2016-05-18 01:48:01,191 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A47CB8FE43B24AD98EF3BA692C8E067A: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:01,194 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A47CB8FE43B24AD98EF3BA692C8E067A: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-local-dir/output/attempt_1463536080937_0001_1_00_000000_0_10119_0/file.out
2016-05-18 01:48:01,197 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:01 Completed running task attempt: attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,197 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:01,198 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536080937_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:01,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536080937_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536081184, Process_Duration=4, Process_End_Time=1463536081188, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:01,199 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536080937_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, taskAttemptId=attempt_1463536080937_0001_1_00_000000_0, creationTime=1463536081135, allocationTime=1463536081137, startTime=1463536081140, finishTime=1463536081199, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:01,200 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,200 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536080937_0001_1_00_000000_0
2016-05-18 01:48:01,201 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:01,201 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,201 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_FINISHED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, taskId=task_1463536080937_0001_1_00_000000, startTime=1463536081140, finishTime=1463536081201, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1463536080937_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536081184, Process_Duration=4, Process_End_Time=1463536081188, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:01,202 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080937_0001_00_000001
2016-05-18 01:48:01,202 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:01,203 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_FINISHED]: vertexName=CC86AB31E5C74E749A360A351BB4001A, vertexId=vertex_1463536080937_0001_1_00, initRequestedTime=1463536081115, initedTime=1463536081130, startRequestedTime=1463536081116, startedTime=1463536081130, finishTime=1463536081203, timeTaken=73, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536081184, Process_Duration=4, Process_End_Time=1463536081188, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536081140, firstTasksToStart=[ task_1463536080937_0001_1_00_000000 ], lastTaskFinishTime=1463536081199, lastTasksToFinish=[ task_1463536080937_0001_1_00_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536080937_0001_1_00_000000 ], longestDurationTasks=[ task_1463536080937_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:01,204 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:01,204 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536080937_0001_00_000001, stoppedTime=1463536081204, exitStatus=0
2016-05-18 01:48:01,204 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] attempt: attempt_1463536080937_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:01,204 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:01,204 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A47CB8FE43B24AD98EF3BA692C8E067A
2016-05-18 01:48:01,204 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CC86AB31E5C74E749A360A351BB4001A, destinationVertexName=A47CB8FE43B24AD98EF3BA692C8E067A. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:01,205 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,204 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,205 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536080937_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:01,206 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_STARTED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, taskId=task_1463536080937_0001_1_01_000000, scheduledTime=1463536081205, launchTime=1463536081205
2016-05-18 01:48:01,206 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,207 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536080937_0001_00_000002, launchTime=1463536081207
2016-05-18 01:48:01,208 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080937_0001_00_000002
2016-05-18 01:48:01,208 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536080937_0001_1_01_000000_0] started. Is using containerId: [container_1463536080937_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:01,208 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0, startTime=1463536081208, containerId=container_1463536080937_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:01,209 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080937_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01 Starting to run new task attempt: attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,210 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,210 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,210 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:01,210 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true, VertexName: A47CB8FE43B24AD98EF3BA692C8E067A, VertexParallelism: 1, TaskAttemptID:attempt_1463536080937_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CC86AB31E5C74E749A360A351BB4001A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=540B8F8DD8A644BEB0623AAD940165E4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:01,210 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:01,211 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,213 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536080937_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:01,213 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080937_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:01,213 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536080937_0001_00_000001, Exiting
2016-05-18 01:48:01,213 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536080937_0001_00_000001
2016-05-18 01:48:01,213 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536080937_0001_00_000001
2016-05-18 01:48:01,214 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536080937_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,216 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536080937_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:01,216 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:01,217 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:01,217 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536080937_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:01,217 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536080937_0001_00_000001 completed successfully
2016-05-18 01:48:01,224 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A47CB8FE43B24AD98EF3BA692C8E067A, ordinal: 1
2016-05-18 01:48:01,224 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A47CB8FE43B24AD98EF3BA692C8E067A, mem on start (mb), free: 153, total: 338, max: 672
2016-05-18 01:48:01,224 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:01,224 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:01,224 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:01,227 INFO output.MROutput (MROutput.java:initialize(421)) - 540B8F8DD8A644BEB0623AAD940165E4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:01,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:01,228 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CC86AB31E5C74E749A360A351BB4001A:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [540B8F8DD8A644BEB0623AAD940165E4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:01,228 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:01,228 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:01,228 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:01,229 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CC86AB31E5C74E749A360A351BB4001A:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [540B8F8DD8A644BEB0623AAD940165E4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:01,229 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:01,229 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CC86AB31E5C74E749A360A351BB4001A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:01,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:01,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:01,230 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CC86AB31E5C74E749A360A351BB4001A: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:01,230 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CC86AB31E5C74E749A360A351BB4001A: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:01,231 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CC86AB31E5C74E749A360A351BB4001A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:01,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CC86AB31E5C74E749A360A351BB4001A
2016-05-18 01:48:01,233 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:01,233 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,233 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC86AB31E5C74E749A360A351BB4001A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:01,235 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536080937_0001_1_00_000000_0_10119} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536081235, TimeTaken=1, Rate=0.70 MB/s
2016-05-18 01:48:01,236 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CC86AB31E5C74E749A360A351BB4001A
2016-05-18 01:48:01,236 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:01,236 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CC86AB31E5C74E749A360A351BB4001A, wasInterrupted=false
2016-05-18 01:48:01,238 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:01,239 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:01,239 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:01,240 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 31B6AC1BEDDD4CB48951155E36ECADE4
2016-05-18 01:48:01,240 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true"], id: 540B8F8DD8A644BEB0623AAD940165E4
2016-05-18 01:48:01,241 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true"]
2016-05-18 01:48:01,241 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001_wd/localmode-local-dir/output/attempt_1463536080937_0001_1_00_000000_0_10119/file.out, len=732, isLocal=true
2016-05-18 01:48:01,241 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 31B6AC1BEDDD4CB48951155E36ECADE4, for 1 inputs
2016-05-18 01:48:01,241 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:01,242 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:01,242 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:01,244 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:01,244 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CC86AB31E5C74E749A360A351BB4001A
2016-05-18 01:48:01,244 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A47CB8FE43B24AD98EF3BA692C8E067A, all 1 inputs ready in: 00:00:00.003
2016-05-18 01:48:01,244 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CC86AB31E5C74E749A360A351BB4001A: Shuffle Runner thread complete
2016-05-18 01:48:01,245 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CC86AB31E5C74E749A360A351BB4001A: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:01,250 INFO output.MROutput (MROutput.java:close(526)) - 540B8F8DD8A644BEB0623AAD940165E4 closed
2016-05-18 01:48:01,250 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536080937_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:01,251 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,251 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A47CB8FE43B24AD98EF3BA692C8E067A, mem on close (mb), free: 149, total: 338, max: 672
2016-05-18 01:48:01,251 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CC86AB31E5C74E749A360A351BB4001A
2016-05-18 01:48:01,251 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC86AB31E5C74E749A360A351BB4001A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:01,251 INFO output.MROutput (MROutput.java:close(526)) - 540B8F8DD8A644BEB0623AAD940165E4 closed
2016-05-18 01:48:01,252 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:01 Completed running task attempt: attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,252 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:01,252 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536080937_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:01,253 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536080937_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536081244, Process_Duration=7, Process_End_Time=1463536081251, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:01,255 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536080937_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,255 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, taskAttemptId=attempt_1463536080937_0001_1_01_000000_0, creationTime=1463536081206, allocationTime=1463536081207, startTime=1463536081208, finishTime=1463536081255, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:01,255 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:01,255 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,255 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536080937_0001_1_01_000000_0
2016-05-18 01:48:01,255 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536080937_0001_00_000002
2016-05-18 01:48:01,256 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:TASK_FINISHED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, taskId=task_1463536080937_0001_1_01_000000, startTime=1463536081208, finishTime=1463536081255, timeTaken=47, status=SUCCEEDED, successfulAttemptID=attempt_1463536080937_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536081244, Process_Duration=7, Process_End_Time=1463536081251, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:01,256 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:01,256 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A]
2016-05-18 01:48:01,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:VERTEX_FINISHED]: vertexName=A47CB8FE43B24AD98EF3BA692C8E067A, vertexId=vertex_1463536080937_0001_1_01, initRequestedTime=1463536081117, initedTime=1463536081122, startRequestedTime=1463536081130, startedTime=1463536081130, finishTime=1463536081256, timeTaken=126, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536081244, Process_Duration=7, Process_End_Time=1463536081251, Read_Duration=4, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536081208, firstTasksToStart=[ task_1463536080937_0001_1_01_000000 ], lastTaskFinishTime=1463536081255, lastTasksToFinish=[ task_1463536080937_0001_1_01_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536080937_0001_1_01_000000 ], longestDurationTasks=[ task_1463536080937_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:01,257 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:01,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536080937_0001_00_000002, stoppedTime=1463536081257, exitStatus=0
2016-05-18 01:48:01,258 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:01,258 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,258 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536080937_0001_1_00 [CC86AB31E5C74E749A360A351BB4001A]
2016-05-18 01:48:01,258 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536080937_0001_1
2016-05-18 01:48:01,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536080937_0001_1
2016-05-18 01:48:01,259 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080937_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:01,259 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 540B8F8DD8A644BEB0623AAD940165E4 for vertex: vertex_1463536080937_0001_1_01 [A47CB8FE43B24AD98EF3BA692C8E067A], outputName: 540B8F8DD8A644BEB0623AAD940165E4
2016-05-18 01:48:01,259 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536080937_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:01,266 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536080937_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:01,266 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536080937_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:01,266 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536080937_0001_00_000002, Exiting
2016-05-18 01:48:01,266 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536080937_0001_00_000002
2016-05-18 01:48:01,266 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536080937_0001_00_000002
2016-05-18 01:48:01,267 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536080937_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:01,267 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536080937_0001_00_000002 completed successfully
2016-05-18 01:48:01,268 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:540B8F8DD8A644BEB0623AAD940165E4 of vertex/vertexGroup:A47CB8FE43B24AD98EF3BA692C8E067A isVertexGroupOutput:false
2016-05-18 01:48:01,268 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,271 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536080937_0001_1, queueSize=0
2016-05-18 01:48:01,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536080937_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536080937_0001_1, startTime=1463536081114, finishTime=1463536081268, timeTaken=154, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=240, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072162428, Process_Duration=11, Process_End_Time=2927072162439, Read_Duration=5, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:01,271 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536080937_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:01,272 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536080937_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:01 Completed Dag: dag_1463536080937_0001_1
2016-05-18 01:48:01,272 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536080937_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:01,272 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536080937_0001_1_post
2016-05-18 01:48:01,272 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,272 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:01,272 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:01,272 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true, with id=dag_1463536080937_0001_1
2016-05-18 01:48:01,273 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10] comprehensivesort/(1/1) ...ve/false_false_false_true, with id=dag_1463536080937_0001_1
2016-05-18 01:48:01,277 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ve/false_false_false_true, applicationId=application_1463536080937_0001
2016-05-18 01:48:01,279 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:01,279 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:01,279 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:01,279 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:01,279 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:01,280 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:01,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:01,280 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:01,281 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:01,281 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:01,281 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:01,281 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:01,281 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:01,282 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:01,286 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4697826F965B479C9036C6F5AC2F749F/DC7BAA9BC00242DEB332A02F0130EA10/.tez/application_1463536080937_0001
2016-05-18 01:48:01,287 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true/_temporary
2016-05-18 01:48:01,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.370
2016-05-18 01:48:01,357 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:01,358 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:01,394 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 01:48:01,406 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.019
2016-05-18 01:48:01,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:01,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:01,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:01,420 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:01,420 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:01,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:01,429 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true
2016-05-18 01:48:01,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:01,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:01,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true"]
2016-05-18 01:48:01,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:01,500 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:01,500 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:01,501 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/true_false_false_true
2016-05-18 01:48:01,501 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:01,514 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:01,518 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:01,518 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:01,518 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:01,527 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001 doesn't exist and is created
2016-05-18 01:48:01,548 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:01,548 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:01,549 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd
2016-05-18 01:48:01,564 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536081518_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:01,564 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:01,564 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:01,566 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:01,566 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:01,566 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:01,566 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:01,567 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:01,569 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:01,569 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:01,569 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:01,569 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:01,569 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:01,569 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:01,570 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:01,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:01,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:01,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:01,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:01,571 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:01,574 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:01,574 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:01,574 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:01,575 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-log-dir
2016-05-18 01:48:01,575 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-log-dir/history.txt.appattempt_1463536081518_0001_000000, maxErrors=10
2016-05-18 01:48:01,575 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:01,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536081518_0001_000000, appSubmitTime=1463536081563, launchTime=1463536081563
2016-05-18 01:48:01,577 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:01,581 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:40741
2016-05-18 01:48:01,581 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536081518
2016-05-18 01:48:01,584 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536081518_0001_000000, startTime=1463536081584
2016-05-18 01:48:01,584 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:01,648 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:01,648 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:01,649 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/true_false_false_true, applicationId=application_1463536081518_0001, dagName=[B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true
2016-05-18 01:48:01,665 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true
2016-05-18 01:48:01,665 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:01,666 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536081518_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-log-dir/dag_1463536081518_0001_1.dot
2016-05-18 01:48:01,667 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-log-dir/dag_1463536081518_0001_1-tez-dag.pb.txt
2016-05-18 01:48:01,673 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536081518_0001_1
2016-05-18 01:48:01,673 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,674 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true
2016-05-18 01:48:01 Running Dag: dag_1463536081518_0001_1
2016-05-18 01:48:01,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536081518_0001_1, submitTime=1463536081665
2016-05-18 01:48:01,690 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001/tez-conf.pb" } size: 40937 timestamp: 1463536081000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:01,690 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,690 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:01,691 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001/tez-conf.pb" } size: 40937 timestamp: 1463536081000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:01,692 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,692 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:01,692 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:01,693 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536081518_0001_1, initTime=1463536081693
2016-05-18 01:48:01,693 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536081518_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:01,694 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:01,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:DAG_STARTED]: dagID=dag_1463536081518_0001_1, startTime=1463536081694
2016-05-18 01:48:01,694 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536081518_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:01,694 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: ED4085CD577448E2B5135FD554B0CEB3 : {B0CB3292E46D4CFD91E977B679132757={InputName=B0CB3292E46D4CFD91E977B679132757}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:01,694 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: B0CB3292E46D4CFD91E977B679132757, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:01,694 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,695 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,695 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,695 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,695 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:01,695 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: B0CB3292E46D4CFD91E977B679132757 on vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,695 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,697 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/true_false_false_true, applicationId=application_1463536081518_0001, dagName=[B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true
2016-05-18 01:48:01,699 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:01,700 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,700 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,700 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input B0CB3292E46D4CFD91E977B679132757 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:01,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536081518_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:01,701 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D], numAdditionalOutputs=1
2016-05-18 01:48:01,701 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=42BFB566693D47C3B860D2F9DDF77873, vertex=vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:01,703 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DD5175E4008047D08331BF2951E5151D:42BFB566693D47C3B860D2F9DDF77873 using oldmapred API
2016-05-18 01:48:01,703 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:01,703 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:01,703 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=42BFB566693D47C3B860D2F9DDF77873, vertexName=DD5175E4008047D08331BF2951E5151D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:01,704 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:01,704 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:01,704 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: B0CB3292E46D4CFD91E977B679132757 on vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,705 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DD5175E4008047D08331BF2951E5151D, vertexId=vertex_1463536081518_0001_1_01, initRequestedTime=1463536081695, initedTime=1463536081701, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:01,705 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:01,706 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:01,706 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {B0CB3292E46D4CFD91E977B679132757=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:01,707 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] parallelism set to 1
2016-05-18 01:48:01,707 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:01,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, vertexId=vertex_1463536081518_0001_1_00, initRequestedTime=1463536081694, initedTime=1463536081707, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:01,708 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] with distanceFromRoot: 0
2016-05-18 01:48:01,708 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:01,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536081518_0001_1_00, startRequestedTime=1463536081695, startedTime=1463536081708
2016-05-18 01:48:01,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536081518_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:01,710 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:01,711 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in ED4085CD577448E2B5135FD554B0CEB3
2016-05-18 01:48:01,711 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,711 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536081518_0001_1_00 for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] numStartedSources: 1 numSources: 1
2016-05-18 01:48:01,711 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] with distanceFromRoot: 1
2016-05-18 01:48:01,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536081518_0001_1_01, startRequestedTime=1463536081711, startedTime=1463536081711
2016-05-18 01:48:01,712 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:01,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_STARTED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, taskId=task_1463536081518_0001_1_00_000000, scheduledTime=1463536081713, launchTime=1463536081713
2016-05-18 01:48:01,713 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DD5175E4008047D08331BF2951E5151D with 0 source tasks and 1 pending tasks
2016-05-18 01:48:01,713 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,714 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: ED4085CD577448E2B5135FD554B0CEB3 in vertex: DD5175E4008047D08331BF2951E5151D numBipartiteSourceTasks: 1
2016-05-18 01:48:01,715 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:01,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536081518_0001_00_000001, launchTime=1463536081717
2016-05-18 01:48:01,717 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536081518_0001_00_000001
2016-05-18 01:48:01,718 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536081518_0001_1_00_000000_0] started. Is using containerId: [container_1463536081518_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:01,718 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0, startTime=1463536081718, containerId=container_1463536081518_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:01,719 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536081518_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01 Starting to run new task attempt: attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,719 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536081518_0001, with dag id: dag_1463536081518_0001_1
2016-05-18 01:48:01,720 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,720 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,720 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:01,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true, VertexName: ED4085CD577448E2B5135FD554B0CEB3, VertexParallelism: 1, TaskAttemptID:attempt_1463536081518_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B0CB3292E46D4CFD91E977B679132757, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DD5175E4008047D08331BF2951E5151D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:01,722 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:01,722 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,723 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536081518_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:01,723 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536081518_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:01,726 INFO input.MRInput (MRInput.java:initialize(440)) - B0CB3292E46D4CFD91E977B679132757 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:01,726 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: B0CB3292E46D4CFD91E977B679132757
2016-05-18 01:48:01,727 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:01,737 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ED4085CD577448E2B5135FD554B0CEB3, ordinal: 0
2016-05-18 01:48:01,737 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED4085CD577448E2B5135FD554B0CEB3, mem on start (mb), free: 231, total: 338, max: 672
2016-05-18 01:48:01,737 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:01,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:01,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:01,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:01,738 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B0CB3292E46D4CFD91E977B679132757:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DD5175E4008047D08331BF2951E5151D:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:01,738 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:01,739 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:01,739 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:01,739 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B0CB3292E46D4CFD91E977B679132757:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DD5175E4008047D08331BF2951E5151D:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:01,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:01,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B0CB3292E46D4CFD91E977B679132757 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:01,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:01,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:01,740 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B0CB3292E46D4CFD91E977B679132757
2016-05-18 01:48:01,741 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:01,741 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,745 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:01,746 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: B0CB3292E46D4CFD91E977B679132757
2016-05-18 01:48:01,746 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: E501CFCB09FB4EBB8C5039C0F6C0C56F
2016-05-18 01:48:01,746 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] E501CFCB09FB4EBB8C5039C0F6C0C56F
2016-05-18 01:48:01,750 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DD5175E4008047D08331BF2951E5151D using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@15426dc8, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@3c35705f, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:01,751 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DD5175E4008047D08331BF2951E5151D: , UsingHashComparator=false
2016-05-18 01:48:01,750 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:01,754 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - B0CB3292E46D4CFD91E977B679132757 initialized RecordReader from event
2016-05-18 01:48:01,765 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:01,766 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:01,766 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DD5175E4008047D08331BF2951E5151D: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:01,766 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DD5175E4008047D08331BF2951E5151D: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:01,767 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:01,767 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ED4085CD577448E2B5135FD554B0CEB3, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:01,772 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,772 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED4085CD577448E2B5135FD554B0CEB3, mem on close (mb), free: 128, total: 338, max: 672
2016-05-18 01:48:01,772 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DD5175E4008047D08331BF2951E5151D: Starting flush of map output
2016-05-18 01:48:01,772 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DD5175E4008047D08331BF2951E5151D: Span0.length = 37, perItem = 24
2016-05-18 01:48:01,774 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DD5175E4008047D08331BF2951E5151D: done sorting span=0, length=37, time=1
2016-05-18 01:48:01,774 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DD5175E4008047D08331BF2951E5151D: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:01,775 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DD5175E4008047D08331BF2951E5151D: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-local-dir/output/attempt_1463536081518_0001_1_00_000000_0_10125_0/file.out
2016-05-18 01:48:01,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:01 Completed running task attempt: attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,780 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:01,780 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536081518_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:01,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536081518_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536081768, Process_Duration=3, Process_End_Time=1463536081771, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:01,782 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536081518_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,782 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,782 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:01,782 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,782 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, taskAttemptId=attempt_1463536081518_0001_1_00_000000_0, creationTime=1463536081713, allocationTime=1463536081716, startTime=1463536081718, finishTime=1463536081782, timeTaken=64, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:01,783 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536081518_0001_1_00_000000_0
2016-05-18 01:48:01,783 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536081518_0001_00_000001
2016-05-18 01:48:01,783 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_FINISHED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, taskId=task_1463536081518_0001_1_00_000000, startTime=1463536081718, finishTime=1463536081783, timeTaken=65, status=SUCCEEDED, successfulAttemptID=attempt_1463536081518_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536081768, Process_Duration=3, Process_End_Time=1463536081771, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:01,784 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:01,784 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,784 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_FINISHED]: vertexName=ED4085CD577448E2B5135FD554B0CEB3, vertexId=vertex_1463536081518_0001_1_00, initRequestedTime=1463536081694, initedTime=1463536081707, startRequestedTime=1463536081695, startedTime=1463536081708, finishTime=1463536081784, timeTaken=76, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536081768, Process_Duration=3, Process_End_Time=1463536081771, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536081718, firstTasksToStart=[ task_1463536081518_0001_1_00_000000 ], lastTaskFinishTime=1463536081782, lastTasksToFinish=[ task_1463536081518_0001_1_00_000000 ], minTaskDuration=64, maxTaskDuration=64, avgTaskDuration=64.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536081518_0001_1_00_000000 ], longestDurationTasks=[ task_1463536081518_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:01,784 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:01,785 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] attempt: attempt_1463536081518_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:01,785 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:01,785 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,788 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536081518_0001_00_000001, stoppedTime=1463536081788, exitStatus=0
2016-05-18 01:48:01,788 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DD5175E4008047D08331BF2951E5151D
2016-05-18 01:48:01,788 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536081518_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:01,788 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=ED4085CD577448E2B5135FD554B0CEB3, destinationVertexName=DD5175E4008047D08331BF2951E5151D. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:01,789 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_STARTED]: vertexName=DD5175E4008047D08331BF2951E5151D, taskId=task_1463536081518_0001_1_01_000000, scheduledTime=1463536081789, launchTime=1463536081789
2016-05-18 01:48:01,790 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536081518_0001_00_000002, launchTime=1463536081791
2016-05-18 01:48:01,792 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536081518_0001_00_000002
2016-05-18 01:48:01,793 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536081518_0001_1_01_000000_0] started. Is using containerId: [container_1463536081518_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:01,793 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DD5175E4008047D08331BF2951E5151D, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0, startTime=1463536081793, containerId=container_1463536081518_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:01,794 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536081518_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,794 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536081518_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:01,794 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536081518_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:01,794 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536081518_0001_00_000001, Exiting
2016-05-18 01:48:01,794 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536081518_0001_00_000001
2016-05-18 01:48:01,795 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536081518_0001_00_000001
2016-05-18 01:48:01,795 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536081518_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:01 Starting to run new task attempt: attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,795 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536081518_0001_00_000001 completed successfully
2016-05-18 01:48:01,795 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,796 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,796 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:01,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true, VertexName: DD5175E4008047D08331BF2951E5151D, VertexParallelism: 1, TaskAttemptID:attempt_1463536081518_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ED4085CD577448E2B5135FD554B0CEB3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=42BFB566693D47C3B860D2F9DDF77873, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:01,796 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:01,797 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,798 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536081518_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,798 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536081518_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:01,800 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:01,802 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:01,812 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DD5175E4008047D08331BF2951E5151D, ordinal: 1
2016-05-18 01:48:01,812 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD5175E4008047D08331BF2951E5151D, mem on start (mb), free: 122, total: 338, max: 672
2016-05-18 01:48:01,813 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:01,813 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:01,813 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:01,815 INFO output.MROutput (MROutput.java:initialize(421)) - 42BFB566693D47C3B860D2F9DDF77873: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:01,815 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:01,815 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[ED4085CD577448E2B5135FD554B0CEB3:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [42BFB566693D47C3B860D2F9DDF77873:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:01,815 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:01,816 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:01,816 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:01,816 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[ED4085CD577448E2B5135FD554B0CEB3:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [42BFB566693D47C3B860D2F9DDF77873:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:01,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:01,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: ED4085CD577448E2B5135FD554B0CEB3 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:01,817 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - ED4085CD577448E2B5135FD554B0CEB3: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:01,817 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - ED4085CD577448E2B5135FD554B0CEB3: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:01,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:01,818 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: ED4085CD577448E2B5135FD554B0CEB3 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:01,819 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:01,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: ED4085CD577448E2B5135FD554B0CEB3
2016-05-18 01:48:01,821 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:01,821 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,823 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ED4085CD577448E2B5135FD554B0CEB3: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:01,826 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:01,827 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536081518_0001_1_00_000000_0_10125} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536081827, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:48:01,827 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: E501CFCB09FB4EBB8C5039C0F6C0C56F
2016-05-18 01:48:01,827 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : ED4085CD577448E2B5135FD554B0CEB3
2016-05-18 01:48:01,827 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true"], id: 42BFB566693D47C3B860D2F9DDF77873
2016-05-18 01:48:01,828 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true"]
2016-05-18 01:48:01,828 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] E501CFCB09FB4EBB8C5039C0F6C0C56F, for 1 inputs
2016-05-18 01:48:01,828 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:01,829 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: ED4085CD577448E2B5135FD554B0CEB3, wasInterrupted=false
2016-05-18 01:48:01,829 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:01,830 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:01,830 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - ED4085CD577448E2B5135FD554B0CEB3: Already shutdown. Ignoring fetch complete
2016-05-18 01:48:01,831 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001_wd/localmode-local-dir/output/attempt_1463536081518_0001_1_00_000000_0_10125/file.out, len=732, isLocal=true
2016-05-18 01:48:01,832 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:01,832 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:01,833 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:01,833 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:01,834 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : ED4085CD577448E2B5135FD554B0CEB3
2016-05-18 01:48:01,834 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DD5175E4008047D08331BF2951E5151D, all 1 inputs ready in: 00:00:00.006
2016-05-18 01:48:01,834 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - ED4085CD577448E2B5135FD554B0CEB3: Shuffle Runner thread complete
2016-05-18 01:48:01,835 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - ED4085CD577448E2B5135FD554B0CEB3: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:01,841 INFO output.MROutput (MROutput.java:close(526)) - 42BFB566693D47C3B860D2F9DDF77873 closed
2016-05-18 01:48:01,842 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536081518_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:01,842 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,843 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD5175E4008047D08331BF2951E5151D, mem on close (mb), free: 119, total: 338, max: 672
2016-05-18 01:48:01,843 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: ED4085CD577448E2B5135FD554B0CEB3
2016-05-18 01:48:01,843 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ED4085CD577448E2B5135FD554B0CEB3: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:01,843 INFO output.MROutput (MROutput.java:close(526)) - 42BFB566693D47C3B860D2F9DDF77873 closed
2016-05-18 01:48:01,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:01 Completed running task attempt: attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,845 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:01,845 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536081518_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:01,846 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536081518_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536081834, Process_Duration=8, Process_End_Time=1463536081842, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:01,846 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536081518_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DD5175E4008047D08331BF2951E5151D, taskAttemptId=attempt_1463536081518_0001_1_01_000000_0, creationTime=1463536081789, allocationTime=1463536081790, startTime=1463536081793, finishTime=1463536081846, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:01,846 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:01,847 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536081518_0001_1_01_000000_0
2016-05-18 01:48:01,847 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:01,847 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,847 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536081518_0001_00_000002
2016-05-18 01:48:01,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:TASK_FINISHED]: vertexName=DD5175E4008047D08331BF2951E5151D, taskId=task_1463536081518_0001_1_01_000000, startTime=1463536081793, finishTime=1463536081847, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463536081518_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536081834, Process_Duration=8, Process_End_Time=1463536081842, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:01,848 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:01,848 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D]
2016-05-18 01:48:01,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:VERTEX_FINISHED]: vertexName=DD5175E4008047D08331BF2951E5151D, vertexId=vertex_1463536081518_0001_1_01, initRequestedTime=1463536081695, initedTime=1463536081701, startRequestedTime=1463536081711, startedTime=1463536081711, finishTime=1463536081848, timeTaken=137, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536081834, Process_Duration=8, Process_End_Time=1463536081842, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536081793, firstTasksToStart=[ task_1463536081518_0001_1_01_000000 ], lastTaskFinishTime=1463536081846, lastTasksToFinish=[ task_1463536081518_0001_1_01_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536081518_0001_1_01_000000 ], longestDurationTasks=[ task_1463536081518_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:01,849 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:01,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536081518_0001_00_000002, stoppedTime=1463536081849, exitStatus=0
2016-05-18 01:48:01,849 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:01,849 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,849 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536081518_0001_1_00 [ED4085CD577448E2B5135FD554B0CEB3]
2016-05-18 01:48:01,849 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536081518_0001_1
2016-05-18 01:48:01,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536081518_0001_1
2016-05-18 01:48:01,850 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536081518_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:01,850 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 42BFB566693D47C3B860D2F9DDF77873 for vertex: vertex_1463536081518_0001_1_01 [DD5175E4008047D08331BF2951E5151D], outputName: 42BFB566693D47C3B860D2F9DDF77873
2016-05-18 01:48:01,850 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536081518_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:01,859 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536081518_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:01,859 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536081518_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:01,859 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536081518_0001_00_000002, Exiting
2016-05-18 01:48:01,859 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536081518_0001_00_000002
2016-05-18 01:48:01,859 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:42BFB566693D47C3B860D2F9DDF77873 of vertex/vertexGroup:DD5175E4008047D08331BF2951E5151D isVertexGroupOutput:false
2016-05-18 01:48:01,859 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536081518_0001_00_000002
2016-05-18 01:48:01,859 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:01,860 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536081518_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:01,860 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536081518_0001_00_000002 completed successfully
2016-05-18 01:48:01,863 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536081518_0001_1, queueSize=0
2016-05-18 01:48:01,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536081518_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536081518_0001_1, startTime=1463536081694, finishTime=1463536081860, timeTaken=166, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=400, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072163602, Process_Duration=11, Process_End_Time=2927072163613, Read_Duration=2, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:01,863 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536081518_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:01,863 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536081518_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:01 Completed Dag: dag_1463536081518_0001_1
2016-05-18 01:48:01,864 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536081518_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:01,864 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536081518_0001_1_post
2016-05-18 01:48:01,864 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:01,864 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:01,864 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:01,864 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true, with id=dag_1463536081518_0001_1
2016-05-18 01:48:01,864 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91] comprehensivesort/(1/1) ...ive/true_false_false_true, with id=dag_1463536081518_0001_1
2016-05-18 01:48:01,872 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/true_false_false_true, applicationId=application_1463536081518_0001
2016-05-18 01:48:01,873 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:01,873 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:01,873 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:01,874 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:01,874 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:01,874 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:01,874 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:01,875 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:01,875 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:01,875 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:01,875 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:01,875 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:01,876 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:01,876 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:01,880 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1616386A5184B6FAFD15713EF3DC45E/D0C16A45D98647F5AB8F63CEAFB17F91/.tez/application_1463536081518_0001
2016-05-18 01:48:01,880 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true/_temporary
2016-05-18 01:48:01,880 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.382
2016-05-18 01:48:01,972 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:01,972 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:02,007 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 01:48:02,014 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 01:48:02,014 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:02,015 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:02,015 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:02,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:02,036 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true
2016-05-18 01:48:02,047 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:02,047 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:02,085 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:02,086 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,086 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true"]
2016-05-18 01:48:02,086 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:02,086 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:02,086 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:02,087 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/true_true_false_true
2016-05-18 01:48:02,087 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:02,100 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:02,104 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:02,105 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:02,105 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:02,113 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001 doesn't exist and is created
2016-05-18 01:48:02,133 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:02,133 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:02,134 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd
2016-05-18 01:48:02,148 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536082104_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:02,149 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:02,149 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:02,151 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:02,151 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:02,151 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:02,151 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:02,151 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:02,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:02,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:02,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:02,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:02,154 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:02,155 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:02,155 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:02,155 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:02,155 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:02,155 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:02,155 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:02,155 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:02,158 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:02,159 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:02,159 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:02,159 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-log-dir
2016-05-18 01:48:02,159 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-log-dir/history.txt.appattempt_1463536082104_0001_000000, maxErrors=10
2016-05-18 01:48:02,160 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:02,160 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536082104_0001_000000, appSubmitTime=1463536082148, launchTime=1463536082148
2016-05-18 01:48:02,165 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:36035
2016-05-18 01:48:02,165 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:02,167 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536082104
2016-05-18 01:48:02,170 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536082104_0001_000000, startTime=1463536082170
2016-05-18 01:48:02,170 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:02,233 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:02,233 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:02,234 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/true_true_false_true, applicationId=application_1463536082104_0001, dagName=[4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true
2016-05-18 01:48:02,252 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true
2016-05-18 01:48:02,252 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:02,253 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536082104_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-log-dir/dag_1463536082104_0001_1.dot
2016-05-18 01:48:02,254 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-log-dir/dag_1463536082104_0001_1-tez-dag.pb.txt
2016-05-18 01:48:02,260 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536082104_0001_1
2016-05-18 01:48:02,260 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,260 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true
2016-05-18 01:48:02 Running Dag: dag_1463536082104_0001_1
2016-05-18 01:48:02,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536082104_0001_1, submitTime=1463536082252
2016-05-18 01:48:02,278 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001/tez-conf.pb" } size: 40937 timestamp: 1463536082000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:02,278 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,278 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:02,279 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001/tez-conf.pb" } size: 40937 timestamp: 1463536082000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:02,279 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,280 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:02,280 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:02,281 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536082104_0001_1, initTime=1463536082280
2016-05-18 01:48:02,281 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082104_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:02,282 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:02,282 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:DAG_STARTED]: dagID=dag_1463536082104_0001_1, startTime=1463536082282
2016-05-18 01:48:02,282 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082104_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:02,282 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C7AB0C8AD0754B5E915887125A0E9333 : {93153794D39F4F98818D31F1D481679F={InputName=93153794D39F4F98818D31F1D481679F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:02,282 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 93153794D39F4F98818D31F1D481679F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:02,282 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,282 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,282 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,283 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,283 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:02,283 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,287 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:02,287 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,287 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,287 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 93153794D39F4F98818D31F1D481679F on vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,288 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/true_true_false_true, applicationId=application_1463536082104_0001, dagName=[4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true
2016-05-18 01:48:02,289 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536082104_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:02,289 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB], numAdditionalOutputs=1
2016-05-18 01:48:02,289 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=12ECE0118F014E5CBE7941F2EAB37869, vertex=vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:02,291 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 93153794D39F4F98818D31F1D481679F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:02,291 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F2819FDCA5C9429E800E0E0B17B20BAB:12ECE0118F014E5CBE7941F2EAB37869 using oldmapred API
2016-05-18 01:48:02,291 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=12ECE0118F014E5CBE7941F2EAB37869, vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:02,292 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, vertexId=vertex_1463536082104_0001_1_01, initRequestedTime=1463536082283, initedTime=1463536082289, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:02,292 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:02,293 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:02,294 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:02,294 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:02,294 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:02,295 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 93153794D39F4F98818D31F1D481679F on vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,296 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:02,296 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {93153794D39F4F98818D31F1D481679F=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:02,296 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] parallelism set to 1
2016-05-18 01:48:02,296 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:02,297 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, vertexId=vertex_1463536082104_0001_1_00, initRequestedTime=1463536082282, initedTime=1463536082297, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:02,297 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] with distanceFromRoot: 0
2016-05-18 01:48:02,297 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:02,297 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536082104_0001_1_00, startRequestedTime=1463536082283, startedTime=1463536082297
2016-05-18 01:48:02,297 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536082104_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:02,297 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:02,298 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536082104_0001_1_00 for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] numStartedSources: 1 numSources: 1
2016-05-18 01:48:02,298 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] with distanceFromRoot: 1
2016-05-18 01:48:02,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536082104_0001_1_01, startRequestedTime=1463536082298, startedTime=1463536082298
2016-05-18 01:48:02,298 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:02,299 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C7AB0C8AD0754B5E915887125A0E9333
2016-05-18 01:48:02,300 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,301 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F2819FDCA5C9429E800E0E0B17B20BAB with 0 source tasks and 1 pending tasks
2016-05-18 01:48:02,302 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_STARTED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, taskId=task_1463536082104_0001_1_00_000000, scheduledTime=1463536082302, launchTime=1463536082302
2016-05-18 01:48:02,303 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,303 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C7AB0C8AD0754B5E915887125A0E9333 in vertex: F2819FDCA5C9429E800E0E0B17B20BAB numBipartiteSourceTasks: 1
2016-05-18 01:48:02,304 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:02,306 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536082104_0001_00_000001, launchTime=1463536082306
2016-05-18 01:48:02,306 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082104_0001_00_000001
2016-05-18 01:48:02,307 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082104_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,307 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536082104_0001_1_00_000000_0] started. Is using containerId: [container_1463536082104_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:02,307 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0, startTime=1463536082307, containerId=container_1463536082104_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:02 Starting to run new task attempt: attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,309 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,309 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,309 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:02,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true, VertexName: C7AB0C8AD0754B5E915887125A0E9333, VertexParallelism: 1, TaskAttemptID:attempt_1463536082104_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=93153794D39F4F98818D31F1D481679F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F2819FDCA5C9429E800E0E0B17B20BAB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:02,310 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:02,310 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,311 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536082104_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:02,311 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536082104_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:02,312 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536082104_0001, with dag id: dag_1463536082104_0001_1
2016-05-18 01:48:02,314 INFO input.MRInput (MRInput.java:initialize(440)) - 93153794D39F4F98818D31F1D481679F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:02,314 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 93153794D39F4F98818D31F1D481679F
2016-05-18 01:48:02,315 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:02,321 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C7AB0C8AD0754B5E915887125A0E9333, ordinal: 0
2016-05-18 01:48:02,321 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7AB0C8AD0754B5E915887125A0E9333, mem on start (mb), free: 185, total: 333, max: 672
2016-05-18 01:48:02,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:02,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:02,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:02,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:02,322 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[93153794D39F4F98818D31F1D481679F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F2819FDCA5C9429E800E0E0B17B20BAB:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:02,323 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:02,323 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:02,323 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:02,323 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[93153794D39F4F98818D31F1D481679F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F2819FDCA5C9429E800E0E0B17B20BAB:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:02,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:02,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 93153794D39F4F98818D31F1D481679F
2016-05-18 01:48:02,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 93153794D39F4F98818D31F1D481679F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:02,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:02,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:02,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:02,325 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,329 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,329 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 93153794D39F4F98818D31F1D481679F
2016-05-18 01:48:02,330 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 9BCFBB242FBB400FA63F77C40306CDE0
2016-05-18 01:48:02,330 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 9BCFBB242FBB400FA63F77C40306CDE0
2016-05-18 01:48:02,331 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:02,332 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 93153794D39F4F98818D31F1D481679F initialized RecordReader from event
2016-05-18 01:48:02,333 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F2819FDCA5C9429E800E0E0B17B20BAB using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4a11008c, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@52e75265, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:02,334 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F2819FDCA5C9429E800E0E0B17B20BAB: , UsingHashComparator=false
2016-05-18 01:48:02,349 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:02,349 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:02,349 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F2819FDCA5C9429E800E0E0B17B20BAB: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:02,349 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F2819FDCA5C9429E800E0E0B17B20BAB: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:02,350 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,350 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C7AB0C8AD0754B5E915887125A0E9333, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:02,354 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,354 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7AB0C8AD0754B5E915887125A0E9333, mem on close (mb), free: 183, total: 434, max: 672
2016-05-18 01:48:02,354 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F2819FDCA5C9429E800E0E0B17B20BAB: Starting flush of map output
2016-05-18 01:48:02,355 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F2819FDCA5C9429E800E0E0B17B20BAB: Span0.length = 37, perItem = 24
2016-05-18 01:48:02,356 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F2819FDCA5C9429E800E0E0B17B20BAB: done sorting span=0, length=37, time=1
2016-05-18 01:48:02,356 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F2819FDCA5C9429E800E0E0B17B20BAB: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:02,357 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F2819FDCA5C9429E800E0E0B17B20BAB: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-local-dir/output/attempt_1463536082104_0001_1_00_000000_0_10131_0/file.out
2016-05-18 01:48:02,361 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:02 Completed running task attempt: attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,362 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:02,362 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536082104_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:02,363 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536082104_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536082350, Process_Duration=4, Process_End_Time=1463536082354, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:02,364 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536082104_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, taskAttemptId=attempt_1463536082104_0001_1_00_000000_0, creationTime=1463536082303, allocationTime=1463536082305, startTime=1463536082307, finishTime=1463536082364, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:02,364 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,364 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:02,364 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536082104_0001_1_00_000000_0
2016-05-18 01:48:02,364 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_FINISHED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, taskId=task_1463536082104_0001_1_00_000000, startTime=1463536082307, finishTime=1463536082364, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463536082104_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536082350, Process_Duration=4, Process_End_Time=1463536082354, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:02,365 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:02,365 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,366 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_FINISHED]: vertexName=C7AB0C8AD0754B5E915887125A0E9333, vertexId=vertex_1463536082104_0001_1_00, initRequestedTime=1463536082282, initedTime=1463536082297, startRequestedTime=1463536082283, startedTime=1463536082297, finishTime=1463536082365, timeTaken=68, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536082350, Process_Duration=4, Process_End_Time=1463536082354, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536082307, firstTasksToStart=[ task_1463536082104_0001_1_00_000000 ], lastTaskFinishTime=1463536082364, lastTasksToFinish=[ task_1463536082104_0001_1_00_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536082104_0001_1_00_000000 ], longestDurationTasks=[ task_1463536082104_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:02,366 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:02,366 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] attempt: attempt_1463536082104_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:02,366 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082104_0001_00_000001
2016-05-18 01:48:02,366 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F2819FDCA5C9429E800E0E0B17B20BAB
2016-05-18 01:48:02,366 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C7AB0C8AD0754B5E915887125A0E9333, destinationVertexName=F2819FDCA5C9429E800E0E0B17B20BAB. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:02,367 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,366 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:02,367 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:02,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_STARTED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, taskId=task_1463536082104_0001_1_01_000000, scheduledTime=1463536082369, launchTime=1463536082369
2016-05-18 01:48:02,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536082104_0001_00_000001, stoppedTime=1463536082369, exitStatus=0
2016-05-18 01:48:02,369 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,370 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536082104_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:02,371 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536082104_0001_00_000002, launchTime=1463536082371
2016-05-18 01:48:02,372 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082104_0001_00_000002
2016-05-18 01:48:02,372 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536082104_0001_1_01_000000_0] started. Is using containerId: [container_1463536082104_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:02,372 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0, startTime=1463536082372, containerId=container_1463536082104_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:02,373 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082104_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02 Starting to run new task attempt: attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,373 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,373 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,373 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:02,373 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true, VertexName: F2819FDCA5C9429E800E0E0B17B20BAB, VertexParallelism: 1, TaskAttemptID:attempt_1463536082104_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C7AB0C8AD0754B5E915887125A0E9333, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=12ECE0118F014E5CBE7941F2EAB37869, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:02,373 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:02,374 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,374 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536082104_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,374 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536082104_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:02,378 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:02,378 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536082104_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:02,381 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082104_0001_00_000001: 15 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:02,381 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536082104_0001_00_000001, Exiting
2016-05-18 01:48:02,382 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536082104_0001_00_000001
2016-05-18 01:48:02,382 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536082104_0001_00_000001
2016-05-18 01:48:02,386 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:02,387 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536082104_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:02,387 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536082104_0001_00_000001 completed successfully
2016-05-18 01:48:02,389 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F2819FDCA5C9429E800E0E0B17B20BAB, ordinal: 1
2016-05-18 01:48:02,389 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2819FDCA5C9429E800E0E0B17B20BAB, mem on start (mb), free: 177, total: 434, max: 672
2016-05-18 01:48:02,389 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:02,389 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:02,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:02,392 INFO output.MROutput (MROutput.java:initialize(421)) - 12ECE0118F014E5CBE7941F2EAB37869: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:02,392 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:02,392 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[12ECE0118F014E5CBE7941F2EAB37869:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C7AB0C8AD0754B5E915887125A0E9333:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:48:02,392 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:02,393 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:02,393 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:02,393 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[12ECE0118F014E5CBE7941F2EAB37869:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C7AB0C8AD0754B5E915887125A0E9333:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:48:02,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:02,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C7AB0C8AD0754B5E915887125A0E9333 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:02,394 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C7AB0C8AD0754B5E915887125A0E9333: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:02,394 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C7AB0C8AD0754B5E915887125A0E9333: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:02,394 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C7AB0C8AD0754B5E915887125A0E9333 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:02,395 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C7AB0C8AD0754B5E915887125A0E9333
2016-05-18 01:48:02,395 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:02,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:02,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:02,396 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,398 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C7AB0C8AD0754B5E915887125A0E9333: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:02,400 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536082104_0001_1_00_000000_0_10131} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536082400, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:48:02,400 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C7AB0C8AD0754B5E915887125A0E9333
2016-05-18 01:48:02,401 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:02,401 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C7AB0C8AD0754B5E915887125A0E9333, wasInterrupted=false
2016-05-18 01:48:02,401 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:02,401 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - C7AB0C8AD0754B5E915887125A0E9333: Already shutdown. Ignoring fetch complete
2016-05-18 01:48:02,402 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:02,402 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 9BCFBB242FBB400FA63F77C40306CDE0
2016-05-18 01:48:02,402 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true"], id: 12ECE0118F014E5CBE7941F2EAB37869
2016-05-18 01:48:02,403 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true"]
2016-05-18 01:48:02,403 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:02,403 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 9BCFBB242FBB400FA63F77C40306CDE0, for 1 inputs
2016-05-18 01:48:02,404 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001_wd/localmode-local-dir/output/attempt_1463536082104_0001_1_00_000000_0_10131/file.out, len=732, isLocal=true
2016-05-18 01:48:02,404 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:02,404 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:02,404 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:02,405 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:02,405 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C7AB0C8AD0754B5E915887125A0E9333
2016-05-18 01:48:02,405 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F2819FDCA5C9429E800E0E0B17B20BAB, all 1 inputs ready in: 00:00:00.002
2016-05-18 01:48:02,405 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C7AB0C8AD0754B5E915887125A0E9333: Shuffle Runner thread complete
2016-05-18 01:48:02,406 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C7AB0C8AD0754B5E915887125A0E9333: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:02,411 INFO output.MROutput (MROutput.java:close(526)) - 12ECE0118F014E5CBE7941F2EAB37869 closed
2016-05-18 01:48:02,411 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536082104_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:02,412 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,412 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2819FDCA5C9429E800E0E0B17B20BAB, mem on close (mb), free: 172, total: 434, max: 672
2016-05-18 01:48:02,412 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C7AB0C8AD0754B5E915887125A0E9333
2016-05-18 01:48:02,412 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C7AB0C8AD0754B5E915887125A0E9333: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:02,412 INFO output.MROutput (MROutput.java:close(526)) - 12ECE0118F014E5CBE7941F2EAB37869 closed
2016-05-18 01:48:02,412 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:02 Completed running task attempt: attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,413 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:02,413 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536082104_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:02,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536082104_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536082405, Process_Duration=7, Process_End_Time=1463536082412, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:02,414 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536082104_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,414 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,414 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:02,414 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, taskAttemptId=attempt_1463536082104_0001_1_01_000000_0, creationTime=1463536082369, allocationTime=1463536082370, startTime=1463536082372, finishTime=1463536082414, timeTaken=42, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:02,414 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536082104_0001_1_01_000000_0
2016-05-18 01:48:02,414 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,415 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082104_0001_00_000002
2016-05-18 01:48:02,415 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:TASK_FINISHED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, taskId=task_1463536082104_0001_1_01_000000, startTime=1463536082372, finishTime=1463536082414, timeTaken=42, status=SUCCEEDED, successfulAttemptID=attempt_1463536082104_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536082405, Process_Duration=7, Process_End_Time=1463536082412, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:02,415 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:02,415 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB]
2016-05-18 01:48:02,416 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:VERTEX_FINISHED]: vertexName=F2819FDCA5C9429E800E0E0B17B20BAB, vertexId=vertex_1463536082104_0001_1_01, initRequestedTime=1463536082283, initedTime=1463536082289, startRequestedTime=1463536082298, startedTime=1463536082298, finishTime=1463536082415, timeTaken=117, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536082405, Process_Duration=7, Process_End_Time=1463536082412, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536082372, firstTasksToStart=[ task_1463536082104_0001_1_01_000000 ], lastTaskFinishTime=1463536082414, lastTasksToFinish=[ task_1463536082104_0001_1_01_000000 ], minTaskDuration=42, maxTaskDuration=42, avgTaskDuration=42.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536082104_0001_1_01_000000 ], longestDurationTasks=[ task_1463536082104_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:02,416 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:02,416 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536082104_0001_00_000002, stoppedTime=1463536082416, exitStatus=0
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536082104_0001_1_00 [C7AB0C8AD0754B5E915887125A0E9333]
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536082104_0001_1
2016-05-18 01:48:02,417 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536082104_0001_1
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082104_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:02,417 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 12ECE0118F014E5CBE7941F2EAB37869 for vertex: vertex_1463536082104_0001_1_01 [F2819FDCA5C9429E800E0E0B17B20BAB], outputName: 12ECE0118F014E5CBE7941F2EAB37869
2016-05-18 01:48:02,418 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536082104_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:02,425 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536082104_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:02,425 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082104_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:02,425 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536082104_0001_00_000002, Exiting
2016-05-18 01:48:02,425 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536082104_0001_00_000002
2016-05-18 01:48:02,426 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536082104_0001_00_000002
2016-05-18 01:48:02,426 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536082104_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:02,426 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536082104_0001_00_000002 completed successfully
2016-05-18 01:48:02,427 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:12ECE0118F014E5CBE7941F2EAB37869 of vertex/vertexGroup:F2819FDCA5C9429E800E0E0B17B20BAB isVertexGroupOutput:false
2016-05-18 01:48:02,427 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:02,429 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536082104_0001_1, queueSize=0
2016-05-18 01:48:02,430 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082104_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536082104_0001_1, startTime=1463536082282, finishTime=1463536082427, timeTaken=145, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=270, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072164755, Process_Duration=11, Process_End_Time=2927072164766, Read_Duration=3, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:02,430 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536082104_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:02,430 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082104_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:02 Completed Dag: dag_1463536082104_0001_1
2016-05-18 01:48:02,430 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536082104_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:02,430 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536082104_0001_1_post
2016-05-18 01:48:02,430 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,431 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:02,431 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:02,431 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true, with id=dag_1463536082104_0001_1
2016-05-18 01:48:02,431 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7] comprehensivesort/(1/1) ...sive/true_true_false_true, with id=dag_1463536082104_0001_1
2016-05-18 01:48:02,439 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/true_true_false_true, applicationId=application_1463536082104_0001
2016-05-18 01:48:02,441 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:02,441 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:02,441 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:02,441 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:02,441 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:02,442 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:02,442 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:02,442 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:02,442 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:02,443 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:02,443 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:02,443 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:02,443 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:02,444 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:02,447 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4AAC2C7C2F9247239B23337025EAE7EC/C59E9E7389834E2B9C5AC0E737FF67D7/.tez/application_1463536082104_0001
2016-05-18 01:48:02,448 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true/_temporary
2016-05-18 01:48:02,449 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.363
2016-05-18 01:48:02,517 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:02,518 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:02,553 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.008
2016-05-18 01:48:02,561 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 01:48:02,562 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:02,562 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:02,562 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:02,577 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:02,585 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true
2016-05-18 01:48:02,660 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:02,661 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,661 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true"]
2016-05-18 01:48:02,661 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:02,663 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:02,663 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:02,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:02,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:02,665 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/false_true_false_true
2016-05-18 01:48:02,665 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:02,677 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:02,682 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:02,682 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:02,682 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:02,691 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001 doesn't exist and is created
2016-05-18 01:48:02,734 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:02,734 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:02,735 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd
2016-05-18 01:48:02,750 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536082682_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:02,750 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:02,750 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:02,751 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:02,751 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:02,752 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:02,752 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:02,753 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:02,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:02,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:02,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:02,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:02,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:02,755 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:02,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:02,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:02,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:02,756 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:02,756 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:02,756 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:02,757 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:02,758 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:02,758 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:02,759 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-log-dir
2016-05-18 01:48:02,759 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-log-dir/history.txt.appattempt_1463536082682_0001_000000, maxErrors=10
2016-05-18 01:48:02,759 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:02,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536082682_0001_000000, appSubmitTime=1463536082749, launchTime=1463536082749
2016-05-18 01:48:02,762 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:02,767 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47785
2016-05-18 01:48:02,768 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536082682
2016-05-18 01:48:02,769 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536082682_0001_000000, startTime=1463536082769
2016-05-18 01:48:02,770 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:02,834 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:02,834 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:02,835 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/false_true_false_true, applicationId=application_1463536082682_0001, dagName=[B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true
2016-05-18 01:48:02,851 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true
2016-05-18 01:48:02,852 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:02,852 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536082682_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-log-dir/dag_1463536082682_0001_1.dot
2016-05-18 01:48:02,853 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-log-dir/dag_1463536082682_0001_1-tez-dag.pb.txt
2016-05-18 01:48:02,859 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536082682_0001_1
2016-05-18 01:48:02,859 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,859 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true
2016-05-18 01:48:02 Running Dag: dag_1463536082682_0001_1
2016-05-18 01:48:02,873 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536082682_0001_1, submitTime=1463536082851
2016-05-18 01:48:02,874 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001/tez-conf.pb" } size: 40937 timestamp: 1463536082000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:02,874 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,875 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:02,875 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001/tez-conf.pb" } size: 40937 timestamp: 1463536082000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:02,876 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,876 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:02,876 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:02,877 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536082682_0001_1, initTime=1463536082876
2016-05-18 01:48:02,877 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082682_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:02,877 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:02,877 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:DAG_STARTED]: dagID=dag_1463536082682_0001_1, startTime=1463536082877
2016-05-18 01:48:02,877 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082682_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:02,877 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A71B04569EA24A128D8863263364372E : {57474D49DE284E1CB494EAF1EE240692={InputName=57474D49DE284E1CB494EAF1EE240692}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:02,877 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 57474D49DE284E1CB494EAF1EE240692, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:02,878 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,878 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,878 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,878 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,878 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/false_true_false_true, applicationId=application_1463536082682_0001, dagName=[B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true
2016-05-18 01:48:02,879 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:02,879 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 57474D49DE284E1CB494EAF1EE240692 on vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,879 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,881 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 57474D49DE284E1CB494EAF1EE240692 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:02,881 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:02,881 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,881 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536082682_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:02,882 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0], numAdditionalOutputs=1
2016-05-18 01:48:02,882 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=EB0BD0367B9B4789BD47C99CBC18C09A, vertex=vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:02,884 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for EBB12A7C248245DA94AD4C4A6187E2A0:EB0BD0367B9B4789BD47C99CBC18C09A using oldmapred API
2016-05-18 01:48:02,884 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=EB0BD0367B9B4789BD47C99CBC18C09A, vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:02,885 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, vertexId=vertex_1463536082682_0001_1_01, initRequestedTime=1463536082879, initedTime=1463536082882, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:02,885 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:02,887 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:02,887 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:02,887 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:02,887 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:02,887 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 57474D49DE284E1CB494EAF1EE240692 on vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,888 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:02,888 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {57474D49DE284E1CB494EAF1EE240692=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:02,889 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] parallelism set to 1
2016-05-18 01:48:02,889 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:02,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A71B04569EA24A128D8863263364372E, vertexId=vertex_1463536082682_0001_1_00, initRequestedTime=1463536082877, initedTime=1463536082889, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:02,890 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] with distanceFromRoot: 0
2016-05-18 01:48:02,890 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:02,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536082682_0001_1_00, startRequestedTime=1463536082879, startedTime=1463536082890
2016-05-18 01:48:02,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536082682_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:02,891 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:02,891 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536082682_0001_1_00 for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] numStartedSources: 1 numSources: 1
2016-05-18 01:48:02,891 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A71B04569EA24A128D8863263364372E
2016-05-18 01:48:02,891 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,891 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] with distanceFromRoot: 1
2016-05-18 01:48:02,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536082682_0001_1_01, startRequestedTime=1463536082891, startedTime=1463536082892
2016-05-18 01:48:02,893 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:02,894 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EBB12A7C248245DA94AD4C4A6187E2A0 with 0 source tasks and 1 pending tasks
2016-05-18 01:48:02,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_STARTED]: vertexName=A71B04569EA24A128D8863263364372E, taskId=task_1463536082682_0001_1_00_000000, scheduledTime=1463536082893, launchTime=1463536082893
2016-05-18 01:48:02,895 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A71B04569EA24A128D8863263364372E in vertex: EBB12A7C248245DA94AD4C4A6187E2A0 numBipartiteSourceTasks: 1
2016-05-18 01:48:02,896 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,896 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:02,898 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536082682_0001_00_000001, launchTime=1463536082898
2016-05-18 01:48:02,899 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082682_0001_00_000001
2016-05-18 01:48:02,899 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082682_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,899 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536082682_0001_1_00_000000_0] started. Is using containerId: [container_1463536082682_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:02,899 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A71B04569EA24A128D8863263364372E, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0, startTime=1463536082899, containerId=container_1463536082682_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:02,901 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536082682_0001, with dag id: dag_1463536082682_0001_1
2016-05-18 01:48:02 Starting to run new task attempt: attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,906 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,906 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,907 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:02,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true, VertexName: A71B04569EA24A128D8863263364372E, VertexParallelism: 1, TaskAttemptID:attempt_1463536082682_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=57474D49DE284E1CB494EAF1EE240692, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EBB12A7C248245DA94AD4C4A6187E2A0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:02,907 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:02,908 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,909 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536082682_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:02,909 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536082682_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:02,910 INFO input.MRInput (MRInput.java:initialize(440)) - 57474D49DE284E1CB494EAF1EE240692 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:02,910 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 57474D49DE284E1CB494EAF1EE240692
2016-05-18 01:48:02,912 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:02,915 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A71B04569EA24A128D8863263364372E, ordinal: 0
2016-05-18 01:48:02,915 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A71B04569EA24A128D8863263364372E, mem on start (mb), free: 243, total: 435, max: 672
2016-05-18 01:48:02,915 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:02,915 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:02,916 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:02,916 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:02,916 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[57474D49DE284E1CB494EAF1EE240692:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [EBB12A7C248245DA94AD4C4A6187E2A0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:02,916 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:02,916 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:02,916 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:02,917 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[57474D49DE284E1CB494EAF1EE240692:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [EBB12A7C248245DA94AD4C4A6187E2A0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:02,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:02,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 57474D49DE284E1CB494EAF1EE240692 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:02,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:02,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:02,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 57474D49DE284E1CB494EAF1EE240692
2016-05-18 01:48:02,918 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:02,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,920 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:02,921 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 57474D49DE284E1CB494EAF1EE240692 initialized RecordReader from event
2016-05-18 01:48:02,922 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,922 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 57474D49DE284E1CB494EAF1EE240692
2016-05-18 01:48:02,922 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: A7064F8982D2432BA56DE18AA25AE9B9
2016-05-18 01:48:02,923 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A7064F8982D2432BA56DE18AA25AE9B9
2016-05-18 01:48:02,924 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EBB12A7C248245DA94AD4C4A6187E2A0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@45e5cad9, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@71689068, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:02,925 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EBB12A7C248245DA94AD4C4A6187E2A0: , UsingHashComparator=false
2016-05-18 01:48:02,937 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:02,937 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:02,937 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EBB12A7C248245DA94AD4C4A6187E2A0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:02,938 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EBB12A7C248245DA94AD4C4A6187E2A0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:02,938 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:02,938 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A71B04569EA24A128D8863263364372E, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:02,942 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,942 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A71B04569EA24A128D8863263364372E, mem on close (mb), free: 141, total: 435, max: 672
2016-05-18 01:48:02,942 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EBB12A7C248245DA94AD4C4A6187E2A0: Starting flush of map output
2016-05-18 01:48:02,943 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EBB12A7C248245DA94AD4C4A6187E2A0: Span0.length = 37, perItem = 24
2016-05-18 01:48:02,944 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EBB12A7C248245DA94AD4C4A6187E2A0: done sorting span=0, length=37, time=1
2016-05-18 01:48:02,944 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EBB12A7C248245DA94AD4C4A6187E2A0: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:02,946 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EBB12A7C248245DA94AD4C4A6187E2A0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-local-dir/output/attempt_1463536082682_0001_1_00_000000_0_10137_0/file.out
2016-05-18 01:48:02,950 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:02 Completed running task attempt: attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,951 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:02,951 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536082682_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:02,951 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536082682_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536082938, Process_Duration=4, Process_End_Time=1463536082942, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:02,952 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536082682_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,952 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:02,953 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:02,953 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,953 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082682_0001_00_000001
2016-05-18 01:48:02,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A71B04569EA24A128D8863263364372E, taskAttemptId=attempt_1463536082682_0001_1_00_000000_0, creationTime=1463536082895, allocationTime=1463536082897, startTime=1463536082899, finishTime=1463536082953, timeTaken=54, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:02,954 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536082682_0001_1_00_000000_0
2016-05-18 01:48:02,954 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_FINISHED]: vertexName=A71B04569EA24A128D8863263364372E, taskId=task_1463536082682_0001_1_00_000000, startTime=1463536082899, finishTime=1463536082954, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463536082682_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536082938, Process_Duration=4, Process_End_Time=1463536082942, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:02,954 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:02,955 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:02,955 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_FINISHED]: vertexName=A71B04569EA24A128D8863263364372E, vertexId=vertex_1463536082682_0001_1_00, initRequestedTime=1463536082877, initedTime=1463536082889, startRequestedTime=1463536082879, startedTime=1463536082890, finishTime=1463536082955, timeTaken=65, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536082938, Process_Duration=4, Process_End_Time=1463536082942, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536082899, firstTasksToStart=[ task_1463536082682_0001_1_00_000000 ], lastTaskFinishTime=1463536082953, lastTasksToFinish=[ task_1463536082682_0001_1_00_000000 ], minTaskDuration=54, maxTaskDuration=54, avgTaskDuration=54.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536082682_0001_1_00_000000 ], longestDurationTasks=[ task_1463536082682_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:02,955 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:02,955 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] attempt: attempt_1463536082682_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:02,955 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:02,955 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:02,956 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EBB12A7C248245DA94AD4C4A6187E2A0
2016-05-18 01:48:02,956 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A71B04569EA24A128D8863263364372E, destinationVertexName=EBB12A7C248245DA94AD4C4A6187E2A0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:02,956 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_STARTED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, taskId=task_1463536082682_0001_1_01_000000, scheduledTime=1463536082956, launchTime=1463536082956
2016-05-18 01:48:02,956 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536082682_0001_00_000001, stoppedTime=1463536082957, exitStatus=0
2016-05-18 01:48:02,957 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536082682_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:02,958 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536082682_0001_00_000002, launchTime=1463536082958
2016-05-18 01:48:02,959 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082682_0001_00_000002
2016-05-18 01:48:02,960 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536082682_0001_1_01_000000_0] started. Is using containerId: [container_1463536082682_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:02,960 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0, startTime=1463536082960, containerId=container_1463536082682_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:02,960 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082682_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:02 Starting to run new task attempt: attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:02,961 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:02,961 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:02,961 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:02,961 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true, VertexName: EBB12A7C248245DA94AD4C4A6187E2A0, VertexParallelism: 1, TaskAttemptID:attempt_1463536082682_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A71B04569EA24A128D8863263364372E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EB0BD0367B9B4789BD47C99CBC18C09A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:02,962 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:02,967 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536082682_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:02,967 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082682_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:02,967 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536082682_0001_00_000001, Exiting
2016-05-18 01:48:02,967 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536082682_0001_00_000001
2016-05-18 01:48:02,967 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536082682_0001_00_000001
2016-05-18 01:48:02,968 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:02,970 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536082682_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:02,970 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536082682_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:02,971 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536082682_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:02,971 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536082682_0001_00_000001 completed successfully
2016-05-18 01:48:02,974 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:02,978 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:02,983 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EBB12A7C248245DA94AD4C4A6187E2A0, ordinal: 1
2016-05-18 01:48:02,984 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EBB12A7C248245DA94AD4C4A6187E2A0, mem on start (mb), free: 135, total: 435, max: 672
2016-05-18 01:48:02,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:02,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:02,985 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:02,996 INFO output.MROutput (MROutput.java:initialize(421)) - EB0BD0367B9B4789BD47C99CBC18C09A: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:02,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:02,997 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A71B04569EA24A128D8863263364372E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [EB0BD0367B9B4789BD47C99CBC18C09A:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:02,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:02,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:02,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:02,997 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A71B04569EA24A128D8863263364372E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [EB0BD0367B9B4789BD47C99CBC18C09A:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:02,997 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:02,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A71B04569EA24A128D8863263364372E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:02,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:02,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:02,999 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A71B04569EA24A128D8863263364372E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:02,999 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A71B04569EA24A128D8863263364372E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:03,000 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A71B04569EA24A128D8863263364372E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:03,001 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A71B04569EA24A128D8863263364372E
2016-05-18 01:48:03,001 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:03,003 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:03,004 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A71B04569EA24A128D8863263364372E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:03,007 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:03,008 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: A7064F8982D2432BA56DE18AA25AE9B9
2016-05-18 01:48:03,008 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true"], id: EB0BD0367B9B4789BD47C99CBC18C09A
2016-05-18 01:48:03,008 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536082682_0001_1_00_000000_0_10137} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536083008, TimeTaken=3, Rate=0.23 MB/s
2016-05-18 01:48:03,008 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true"]
2016-05-18 01:48:03,008 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A71B04569EA24A128D8863263364372E
2016-05-18 01:48:03,009 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A7064F8982D2432BA56DE18AA25AE9B9, for 1 inputs
2016-05-18 01:48:03,009 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:03,009 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A71B04569EA24A128D8863263364372E, wasInterrupted=false
2016-05-18 01:48:03,010 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:03,011 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:03,012 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001_wd/localmode-local-dir/output/attempt_1463536082682_0001_1_00_000000_0_10137/file.out, len=732, isLocal=true
2016-05-18 01:48:03,012 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:03,012 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:03,012 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:03,013 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:03,013 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A71B04569EA24A128D8863263364372E
2016-05-18 01:48:03,013 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EBB12A7C248245DA94AD4C4A6187E2A0, all 1 inputs ready in: 00:00:00.004
2016-05-18 01:48:03,013 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A71B04569EA24A128D8863263364372E: Shuffle Runner thread complete
2016-05-18 01:48:03,014 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A71B04569EA24A128D8863263364372E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:03,020 INFO output.MROutput (MROutput.java:close(526)) - EB0BD0367B9B4789BD47C99CBC18C09A closed
2016-05-18 01:48:03,021 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536082682_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:03,022 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:03,022 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EBB12A7C248245DA94AD4C4A6187E2A0, mem on close (mb), free: 130, total: 435, max: 672
2016-05-18 01:48:03,022 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A71B04569EA24A128D8863263364372E
2016-05-18 01:48:03,022 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A71B04569EA24A128D8863263364372E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:03,023 INFO output.MROutput (MROutput.java:close(526)) - EB0BD0367B9B4789BD47C99CBC18C09A closed
2016-05-18 01:48:03,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:03 Completed running task attempt: attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:03,025 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:03,025 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536082682_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:03,026 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536082682_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536083013, Process_Duration=9, Process_End_Time=1463536083022, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:03,027 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536082682_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,027 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,027 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:03,027 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,027 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, taskAttemptId=attempt_1463536082682_0001_1_01_000000_0, creationTime=1463536082956, allocationTime=1463536082958, startTime=1463536082960, finishTime=1463536083027, timeTaken=67, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:03,027 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536082682_0001_1_01_000000_0
2016-05-18 01:48:03,028 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:TASK_FINISHED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, taskId=task_1463536082682_0001_1_01_000000, startTime=1463536082960, finishTime=1463536083027, timeTaken=67, status=SUCCEEDED, successfulAttemptID=attempt_1463536082682_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536083013, Process_Duration=9, Process_End_Time=1463536083022, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:03,028 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:03,028 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0]
2016-05-18 01:48:03,029 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:VERTEX_FINISHED]: vertexName=EBB12A7C248245DA94AD4C4A6187E2A0, vertexId=vertex_1463536082682_0001_1_01, initRequestedTime=1463536082879, initedTime=1463536082882, startRequestedTime=1463536082891, startedTime=1463536082892, finishTime=1463536083028, timeTaken=136, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536083013, Process_Duration=9, Process_End_Time=1463536083022, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536082960, firstTasksToStart=[ task_1463536082682_0001_1_01_000000 ], lastTaskFinishTime=1463536083027, lastTasksToFinish=[ task_1463536082682_0001_1_01_000000 ], minTaskDuration=67, maxTaskDuration=67, avgTaskDuration=67.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536082682_0001_1_01_000000 ], longestDurationTasks=[ task_1463536082682_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:03,029 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:03,030 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536082682_0001_00_000002, stoppedTime=1463536083030, exitStatus=0
2016-05-18 01:48:03,030 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:03,030 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:03,030 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536082682_0001_1_00 [A71B04569EA24A128D8863263364372E]
2016-05-18 01:48:03,030 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536082682_0001_1
2016-05-18 01:48:03,030 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536082682_0001_1
2016-05-18 01:48:03,030 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082682_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:03,031 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536082682_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:03,031 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: EB0BD0367B9B4789BD47C99CBC18C09A for vertex: vertex_1463536082682_0001_1_01 [EBB12A7C248245DA94AD4C4A6187E2A0], outputName: EB0BD0367B9B4789BD47C99CBC18C09A
2016-05-18 01:48:03,031 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536082682_0001_00_000002
2016-05-18 01:48:03,031 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536082682_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:03,031 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536082682_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:03,032 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536082682_0001_00_000002, Exiting
2016-05-18 01:48:03,032 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536082682_0001_00_000002
2016-05-18 01:48:03,032 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536082682_0001_00_000002
2016-05-18 01:48:03,032 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536082682_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:03,032 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536082682_0001_00_000002 completed successfully
2016-05-18 01:48:03,042 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:EB0BD0367B9B4789BD47C99CBC18C09A of vertex/vertexGroup:EBB12A7C248245DA94AD4C4A6187E2A0 isVertexGroupOutput:false
2016-05-18 01:48:03,042 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:03,044 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536082682_0001_1, queueSize=0
2016-05-18 01:48:03,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536082682_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536082682_0001_1, startTime=1463536082877, finishTime=1463536083042, timeTaken=165, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=390, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072165951, Process_Duration=13, Process_End_Time=2927072165964, Read_Duration=5, Tuples_Read=74, Tuples_Written=74, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:03,045 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536082682_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:03,045 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536082682_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:03 Completed Dag: dag_1463536082682_0001_1
2016-05-18 01:48:03,045 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536082682_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:03,045 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536082682_0001_1_post
2016-05-18 01:48:03,046 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,046 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:03,046 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:03,046 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true, with id=dag_1463536082682_0001_1
2016-05-18 01:48:03,046 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E] comprehensivesort/(1/1) ...ive/false_true_false_true, with id=dag_1463536082682_0001_1
2016-05-18 01:48:03,054 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/false_true_false_true, applicationId=application_1463536082682_0001
2016-05-18 01:48:03,056 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:03,056 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:03,056 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:03,056 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:03,056 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:03,056 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:03,058 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:03,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:03,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:03,059 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:03,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:03,059 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:03,060 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:03,061 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:03,064 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B0DDA3411FB4479B8D4E4CCBFCA580C1/DE131501237449D7A92A32FA55F3763E/.tez/application_1463536082682_0001
2016-05-18 01:48:03,064 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true/_temporary
2016-05-18 01:48:03,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.404
2016-05-18 01:48:03,162 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:03,163 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:03,199 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 01:48:03,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 01:48:03,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:03,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:03,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:03,219 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:03,225 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true
2016-05-18 01:48:03,272 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:03,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:03,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true"]
2016-05-18 01:48:03,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:03,274 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:03,274 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:03,275 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...ive/false_false_true_true
2016-05-18 01:48:03,275 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:03,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:03,293 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:03,294 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:03,294 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:03,304 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001 doesn't exist and is created
2016-05-18 01:48:03,306 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:03,306 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:03,324 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:03,324 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:03,324 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd
2016-05-18 01:48:03,339 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536083293_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:03,340 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:03,340 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:03,342 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:03,342 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:03,342 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:03,342 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:03,344 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:03,345 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:03,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:03,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:03,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:03,347 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:03,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:03,348 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:03,350 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:03,351 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:03,351 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:03,351 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-log-dir
2016-05-18 01:48:03,352 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-log-dir/history.txt.appattempt_1463536083293_0001_000000, maxErrors=10
2016-05-18 01:48:03,352 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:03,352 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536083293_0001_000000, appSubmitTime=1463536083339, launchTime=1463536083339
2016-05-18 01:48:03,355 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:03,357 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:49578
2016-05-18 01:48:03,357 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536083293
2016-05-18 01:48:03,362 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536083293_0001_000000, startTime=1463536083362
2016-05-18 01:48:03,362 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:03,424 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:03,424 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:03,425 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ive/false_false_true_true, applicationId=application_1463536083293_0001, dagName=[F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true
2016-05-18 01:48:03,440 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true
2016-05-18 01:48:03,440 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:03,441 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536083293_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-log-dir/dag_1463536083293_0001_1.dot
2016-05-18 01:48:03,441 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-log-dir/dag_1463536083293_0001_1-tez-dag.pb.txt
2016-05-18 01:48:03,447 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536083293_0001_1
2016-05-18 01:48:03,447 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,448 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true
2016-05-18 01:48:03 Running Dag: dag_1463536083293_0001_1
2016-05-18 01:48:03,463 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536083293_0001_1, submitTime=1463536083440
2016-05-18 01:48:03,465 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001/tez-conf.pb" } size: 40937 timestamp: 1463536083000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:03,465 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,465 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:03,467 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001/tez-conf.pb" } size: 40937 timestamp: 1463536083000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:03,467 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,467 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:03,468 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:03,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536083293_0001_1, initTime=1463536083468
2016-05-18 01:48:03,469 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083293_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:03,469 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:03,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:DAG_STARTED]: dagID=dag_1463536083293_0001_1, startTime=1463536083469
2016-05-18 01:48:03,469 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ive/false_false_true_true, applicationId=application_1463536083293_0001, dagName=[F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true
2016-05-18 01:48:03,469 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083293_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A7BAD121FD0C4C78AFA6B65755BA16D0 : {F9005D53709742DFA806D8A8C95C6203={InputName=F9005D53709742DFA806D8A8C95C6203}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: F9005D53709742DFA806D8A8C95C6203, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,470 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,472 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:03,472 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,473 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: F9005D53709742DFA806D8A8C95C6203 on vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,475 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input F9005D53709742DFA806D8A8C95C6203 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:03,475 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:03,476 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,476 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,476 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536083293_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:03,477 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C], numAdditionalOutputs=1
2016-05-18 01:48:03,477 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=8CB031ACB9BA4A0FBDB12E7167F80C6A, vertex=vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:03,479 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:03,479 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:03,479 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:03,479 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:03,479 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CD404AE93E7A49008F2F6C71D0E2189C:8CB031ACB9BA4A0FBDB12E7167F80C6A using oldmapred API
2016-05-18 01:48:03,479 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=8CB031ACB9BA4A0FBDB12E7167F80C6A, vertexName=CD404AE93E7A49008F2F6C71D0E2189C, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:03,481 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, vertexId=vertex_1463536083293_0001_1_01, initRequestedTime=1463536083472, initedTime=1463536083477, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:03,481 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:03,481 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: F9005D53709742DFA806D8A8C95C6203 on vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,482 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:03,482 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {F9005D53709742DFA806D8A8C95C6203=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:03,483 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] parallelism set to 1
2016-05-18 01:48:03,483 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:03,484 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, vertexId=vertex_1463536083293_0001_1_00, initRequestedTime=1463536083470, initedTime=1463536083484, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:03,485 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] with distanceFromRoot: 0
2016-05-18 01:48:03,485 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:03,485 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536083293_0001_1_00, startRequestedTime=1463536083472, startedTime=1463536083485
2016-05-18 01:48:03,486 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536083293_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:03,486 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:03,486 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536083293_0001_1_00 for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] numStartedSources: 1 numSources: 1
2016-05-18 01:48:03,487 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] with distanceFromRoot: 1
2016-05-18 01:48:03,487 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A7BAD121FD0C4C78AFA6B65755BA16D0
2016-05-18 01:48:03,487 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,488 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536083293_0001_1_01, startRequestedTime=1463536083486, startedTime=1463536083487
2016-05-18 01:48:03,488 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:03,488 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_STARTED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, taskId=task_1463536083293_0001_1_00_000000, scheduledTime=1463536083488, launchTime=1463536083488
2016-05-18 01:48:03,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536083293_0001, with dag id: dag_1463536083293_0001_1
2016-05-18 01:48:03,489 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,490 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:03,492 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CD404AE93E7A49008F2F6C71D0E2189C with 0 source tasks and 1 pending tasks
2016-05-18 01:48:03,493 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A7BAD121FD0C4C78AFA6B65755BA16D0 in vertex: CD404AE93E7A49008F2F6C71D0E2189C numBipartiteSourceTasks: 1
2016-05-18 01:48:03,495 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536083293_0001_00_000001, launchTime=1463536083494
2016-05-18 01:48:03,495 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083293_0001_00_000001
2016-05-18 01:48:03,496 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083293_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,496 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536083293_0001_1_00_000000_0] started. Is using containerId: [container_1463536083293_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:03,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0, startTime=1463536083496, containerId=container_1463536083293_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:03 Starting to run new task attempt: attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,497 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,497 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,497 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:03,497 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true, VertexName: A7BAD121FD0C4C78AFA6B65755BA16D0, VertexParallelism: 1, TaskAttemptID:attempt_1463536083293_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F9005D53709742DFA806D8A8C95C6203, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CD404AE93E7A49008F2F6C71D0E2189C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:03,498 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:03,498 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,498 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536083293_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:03,499 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536083293_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:03,500 INFO input.MRInput (MRInput.java:initialize(440)) - F9005D53709742DFA806D8A8C95C6203 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:03,500 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: F9005D53709742DFA806D8A8C95C6203
2016-05-18 01:48:03,503 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:03,506 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A7BAD121FD0C4C78AFA6B65755BA16D0, ordinal: 0
2016-05-18 01:48:03,506 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7BAD121FD0C4C78AFA6B65755BA16D0, mem on start (mb), free: 184, total: 425, max: 672
2016-05-18 01:48:03,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:03,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:03,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:03,507 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:03,507 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F9005D53709742DFA806D8A8C95C6203:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CD404AE93E7A49008F2F6C71D0E2189C:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:03,507 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:03,507 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:03,508 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:03,508 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F9005D53709742DFA806D8A8C95C6203:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CD404AE93E7A49008F2F6C71D0E2189C:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:03,508 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:03,508 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F9005D53709742DFA806D8A8C95C6203
2016-05-18 01:48:03,508 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F9005D53709742DFA806D8A8C95C6203 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:03,509 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:03,509 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:03,509 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:03,509 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,511 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:03,511 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - F9005D53709742DFA806D8A8C95C6203 initialized RecordReader from event
2016-05-18 01:48:03,513 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:03,513 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: F9005D53709742DFA806D8A8C95C6203
2016-05-18 01:48:03,513 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 96906BD0461D4C71B3C6A2F1AF28A968
2016-05-18 01:48:03,513 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 96906BD0461D4C71B3C6A2F1AF28A968
2016-05-18 01:48:03,515 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CD404AE93E7A49008F2F6C71D0E2189C using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@12dc1a0d, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@30325527, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:03,515 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CD404AE93E7A49008F2F6C71D0E2189C: , UsingHashComparator=false
2016-05-18 01:48:03,627 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:03,627 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:03,627 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CD404AE93E7A49008F2F6C71D0E2189C: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:03,628 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CD404AE93E7A49008F2F6C71D0E2189C: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:03,628 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:03,628 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A7BAD121FD0C4C78AFA6B65755BA16D0, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:03,632 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,632 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A7BAD121FD0C4C78AFA6B65755BA16D0, mem on close (mb), free: 182, total: 525, max: 672
2016-05-18 01:48:03,632 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CD404AE93E7A49008F2F6C71D0E2189C: Starting flush of map output
2016-05-18 01:48:03,632 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CD404AE93E7A49008F2F6C71D0E2189C: Span0.length = 37, perItem = 24
2016-05-18 01:48:03,633 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CD404AE93E7A49008F2F6C71D0E2189C: done sorting span=0, length=37, time=0
2016-05-18 01:48:03,634 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CD404AE93E7A49008F2F6C71D0E2189C: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:03,635 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CD404AE93E7A49008F2F6C71D0E2189C: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-local-dir/output/attempt_1463536083293_0001_1_00_000000_0_10143_0/file.out
2016-05-18 01:48:03,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:03 Completed running task attempt: attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,639 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:03,639 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536083293_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:03,639 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536083293_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536083628, Process_Duration=4, Process_End_Time=1463536083632, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:03,640 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536083293_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,640 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, taskAttemptId=attempt_1463536083293_0001_1_00_000000_0, creationTime=1463536083489, allocationTime=1463536083494, startTime=1463536083496, finishTime=1463536083640, timeTaken=144, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:03,640 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,640 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:03,640 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,640 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536083293_0001_1_00_000000_0
2016-05-18 01:48:03,641 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_FINISHED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, taskId=task_1463536083293_0001_1_00_000000, startTime=1463536083496, finishTime=1463536083641, timeTaken=145, status=SUCCEEDED, successfulAttemptID=attempt_1463536083293_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536083628, Process_Duration=4, Process_End_Time=1463536083632, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:03,641 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:03,642 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,642 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_FINISHED]: vertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, vertexId=vertex_1463536083293_0001_1_00, initRequestedTime=1463536083470, initedTime=1463536083484, startRequestedTime=1463536083472, startedTime=1463536083485, finishTime=1463536083642, timeTaken=157, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536083628, Process_Duration=4, Process_End_Time=1463536083632, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536083496, firstTasksToStart=[ task_1463536083293_0001_1_00_000000 ], lastTaskFinishTime=1463536083640, lastTasksToFinish=[ task_1463536083293_0001_1_00_000000 ], minTaskDuration=144, maxTaskDuration=144, avgTaskDuration=144.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536083293_0001_1_00_000000 ], longestDurationTasks=[ task_1463536083293_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:03,643 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:03,643 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] attempt: attempt_1463536083293_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:03,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536083293_0001_00_000001, stoppedTime=1463536083643, exitStatus=0
2016-05-18 01:48:03,643 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CD404AE93E7A49008F2F6C71D0E2189C
2016-05-18 01:48:03,643 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, destinationVertexName=CD404AE93E7A49008F2F6C71D0E2189C. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:03,643 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,643 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:03,644 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:03,644 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536083293_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:03,644 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_STARTED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, taskId=task_1463536083293_0001_1_01_000000, scheduledTime=1463536083644, launchTime=1463536083644
2016-05-18 01:48:03,644 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,643 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083293_0001_00_000001
2016-05-18 01:48:03,645 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536083293_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:03,645 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083293_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:03,646 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536083293_0001_00_000001, Exiting
2016-05-18 01:48:03,646 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536083293_0001_00_000001
2016-05-18 01:48:03,646 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536083293_0001_00_000001
2016-05-18 01:48:03,646 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536083293_0001_00_000002, launchTime=1463536083646
2016-05-18 01:48:03,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536083293_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:03,647 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536083293_0001_00_000001 completed successfully
2016-05-18 01:48:03,647 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083293_0001_00_000002
2016-05-18 01:48:03,648 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083293_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,648 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536083293_0001_1_01_000000_0] started. Is using containerId: [container_1463536083293_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:03,648 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0, startTime=1463536083648, containerId=container_1463536083293_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:03 Starting to run new task attempt: attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,648 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,648 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,649 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:03,649 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true, VertexName: CD404AE93E7A49008F2F6C71D0E2189C, VertexParallelism: 1, TaskAttemptID:attempt_1463536083293_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A7BAD121FD0C4C78AFA6B65755BA16D0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8CB031ACB9BA4A0FBDB12E7167F80C6A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:03,649 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:03,649 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,650 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536083293_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,650 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536083293_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:03,653 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:03,654 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:03,659 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CD404AE93E7A49008F2F6C71D0E2189C, ordinal: 1
2016-05-18 01:48:03,659 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CD404AE93E7A49008F2F6C71D0E2189C, mem on start (mb), free: 175, total: 525, max: 672
2016-05-18 01:48:03,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:03,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:03,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:03,662 INFO output.MROutput (MROutput.java:initialize(421)) - 8CB031ACB9BA4A0FBDB12E7167F80C6A: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:03,662 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:03,662 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[8CB031ACB9BA4A0FBDB12E7167F80C6A:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A7BAD121FD0C4C78AFA6B65755BA16D0:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:48:03,663 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:03,663 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:03,663 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:03,663 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[8CB031ACB9BA4A0FBDB12E7167F80C6A:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A7BAD121FD0C4C78AFA6B65755BA16D0:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:48:03,663 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:03,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A7BAD121FD0C4C78AFA6B65755BA16D0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:03,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:03,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:03,664 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A7BAD121FD0C4C78AFA6B65755BA16D0: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:03,665 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A7BAD121FD0C4C78AFA6B65755BA16D0: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:03,666 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A7BAD121FD0C4C78AFA6B65755BA16D0 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:03,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A7BAD121FD0C4C78AFA6B65755BA16D0
2016-05-18 01:48:03,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:03,668 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,668 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A7BAD121FD0C4C78AFA6B65755BA16D0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:03,671 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536083293_0001_1_00_000000_0_10143} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536083671, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:48:03,671 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A7BAD121FD0C4C78AFA6B65755BA16D0
2016-05-18 01:48:03,671 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:03,672 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A7BAD121FD0C4C78AFA6B65755BA16D0, wasInterrupted=false
2016-05-18 01:48:03,672 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:03,673 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:03,673 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001_wd/localmode-local-dir/output/attempt_1463536083293_0001_1_00_000000_0_10143/file.out, len=732, isLocal=true
2016-05-18 01:48:03,674 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:03,674 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:03,674 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:03,674 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:03,674 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A7BAD121FD0C4C78AFA6B65755BA16D0
2016-05-18 01:48:03,674 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A7BAD121FD0C4C78AFA6B65755BA16D0: Shuffle Runner thread complete
2016-05-18 01:48:03,675 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:03,675 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 96906BD0461D4C71B3C6A2F1AF28A968
2016-05-18 01:48:03,675 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true"], id: 8CB031ACB9BA4A0FBDB12E7167F80C6A
2016-05-18 01:48:03,675 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true"]
2016-05-18 01:48:03,676 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 96906BD0461D4C71B3C6A2F1AF28A968, for 1 inputs
2016-05-18 01:48:03,676 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CD404AE93E7A49008F2F6C71D0E2189C, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:03,676 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A7BAD121FD0C4C78AFA6B65755BA16D0: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:03,681 INFO output.MROutput (MROutput.java:close(526)) - 8CB031ACB9BA4A0FBDB12E7167F80C6A closed
2016-05-18 01:48:03,681 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536083293_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:03,681 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,682 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CD404AE93E7A49008F2F6C71D0E2189C, mem on close (mb), free: 172, total: 525, max: 672
2016-05-18 01:48:03,682 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A7BAD121FD0C4C78AFA6B65755BA16D0
2016-05-18 01:48:03,682 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A7BAD121FD0C4C78AFA6B65755BA16D0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:03,682 INFO output.MROutput (MROutput.java:close(526)) - 8CB031ACB9BA4A0FBDB12E7167F80C6A closed
2016-05-18 01:48:03,682 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:03 Completed running task attempt: attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,683 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:03,683 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536083293_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:03,683 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536083293_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536083676, Process_Duration=5, Process_End_Time=1463536083681, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:03,684 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536083293_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,684 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:03,684 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:03,684 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, taskAttemptId=attempt_1463536083293_0001_1_01_000000_0, creationTime=1463536083644, allocationTime=1463536083645, startTime=1463536083648, finishTime=1463536083684, timeTaken=36, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:03,685 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,685 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536083293_0001_1_01_000000_0
2016-05-18 01:48:03,685 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083293_0001_00_000002
2016-05-18 01:48:03,685 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:TASK_FINISHED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, taskId=task_1463536083293_0001_1_01_000000, startTime=1463536083648, finishTime=1463536083685, timeTaken=37, status=SUCCEEDED, successfulAttemptID=attempt_1463536083293_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536083676, Process_Duration=5, Process_End_Time=1463536083681, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:03,686 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:03,686 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C]
2016-05-18 01:48:03,686 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:VERTEX_FINISHED]: vertexName=CD404AE93E7A49008F2F6C71D0E2189C, vertexId=vertex_1463536083293_0001_1_01, initRequestedTime=1463536083472, initedTime=1463536083477, startRequestedTime=1463536083486, startedTime=1463536083487, finishTime=1463536083686, timeTaken=199, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536083676, Process_Duration=5, Process_End_Time=1463536083681, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536083648, firstTasksToStart=[ task_1463536083293_0001_1_01_000000 ], lastTaskFinishTime=1463536083684, lastTasksToFinish=[ task_1463536083293_0001_1_01_000000 ], minTaskDuration=36, maxTaskDuration=36, avgTaskDuration=36.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536083293_0001_1_01_000000 ], longestDurationTasks=[ task_1463536083293_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:03,686 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:03,687 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:03,687 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:03,687 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536083293_0001_1_00 [A7BAD121FD0C4C78AFA6B65755BA16D0]
2016-05-18 01:48:03,687 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536083293_0001_1
2016-05-18 01:48:03,687 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536083293_0001_1
2016-05-18 01:48:03,687 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083293_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:03,688 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 8CB031ACB9BA4A0FBDB12E7167F80C6A for vertex: vertex_1463536083293_0001_1_01 [CD404AE93E7A49008F2F6C71D0E2189C], outputName: 8CB031ACB9BA4A0FBDB12E7167F80C6A
2016-05-18 01:48:03,690 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536083293_0001_00_000002, stoppedTime=1463536083690, exitStatus=0
2016-05-18 01:48:03,690 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536083293_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:03,695 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536083293_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:03,695 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083293_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:03,696 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536083293_0001_00_000002, Exiting
2016-05-18 01:48:03,696 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536083293_0001_00_000002
2016-05-18 01:48:03,696 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536083293_0001_00_000002
2016-05-18 01:48:03,696 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536083293_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:03,696 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536083293_0001_00_000002 completed successfully
2016-05-18 01:48:03,700 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:8CB031ACB9BA4A0FBDB12E7167F80C6A of vertex/vertexGroup:CD404AE93E7A49008F2F6C71D0E2189C isVertexGroupOutput:false
2016-05-18 01:48:03,700 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:03,702 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536083293_0001_1, queueSize=0
2016-05-18 01:48:03,702 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083293_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536083293_0001_1, startTime=1463536083469, finishTime=1463536083700, timeTaken=231, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=350, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072167304, Process_Duration=9, Process_End_Time=2927072167313, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:03,702 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536083293_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:03,702 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083293_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:03 Completed Dag: dag_1463536083293_0001_1
2016-05-18 01:48:03,702 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536083293_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:03,702 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536083293_0001_1_post
2016-05-18 01:48:03,702 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:03,703 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:03,703 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:03,703 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true, with id=dag_1463536083293_0001_1
2016-05-18 01:48:03,703 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40] comprehensivesort/(1/1) ...ive/false_false_true_true, with id=dag_1463536083293_0001_1
2016-05-18 01:48:03,715 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ive/false_false_true_true, applicationId=application_1463536083293_0001
2016-05-18 01:48:03,716 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:03,717 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:03,717 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:03,717 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:03,717 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:03,717 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:03,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:03,718 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:03,718 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:03,718 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:03,718 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:03,718 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:03,719 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:03,719 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:03,722 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/F1607C21FA504256A0C9072E4AAFB1C1/A0366BD5FD07421BA07840A5D9F9FE40/.tez/application_1463536083293_0001
2016-05-18 01:48:03,722 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true/_temporary
2016-05-18 01:48:03,723 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.450
2016-05-18 01:48:03,793 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:03,793 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:03,826 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 01:48:03,834 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 01:48:03,834 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:03,834 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:03,835 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:03,849 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:03,855 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true
2016-05-18 01:48:03,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:03,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:03,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true"]
2016-05-18 01:48:03,907 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:03,907 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:03,907 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:03,908 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/true_false_true_true
2016-05-18 01:48:03,908 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:03,919 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:03,919 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:03,921 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:03,925 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:03,926 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:03,926 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:03,934 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001 doesn't exist and is created
2016-05-18 01:48:03,953 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:03,953 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:03,954 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd
2016-05-18 01:48:03,991 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536083926_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:03,991 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:03,991 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:03,992 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:03,992 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:03,993 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:03,993 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:03,993 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:03,994 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:03,994 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:03,994 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:03,994 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:03,995 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:03,995 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:03,995 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:03,996 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:03,997 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:03,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:03,998 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-log-dir
2016-05-18 01:48:03,998 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-log-dir/history.txt.appattempt_1463536083926_0001_000000, maxErrors=10
2016-05-18 01:48:03,998 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:03,998 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536083926_0001_000000, appSubmitTime=1463536083990, launchTime=1463536083990
2016-05-18 01:48:04,001 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:04,005 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:46525
2016-05-18 01:48:04,005 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536083926
2016-05-18 01:48:04,007 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536083926_0001_000000, startTime=1463536084007
2016-05-18 01:48:04,007 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:04,054 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:04,054 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:04,054 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/true_false_true_true, applicationId=application_1463536083926_0001, dagName=[BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true
2016-05-18 01:48:04,073 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true
2016-05-18 01:48:04,074 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:04,074 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536083926_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-log-dir/dag_1463536083926_0001_1.dot
2016-05-18 01:48:04,075 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-log-dir/dag_1463536083926_0001_1-tez-dag.pb.txt
2016-05-18 01:48:04,084 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536083926_0001_1
2016-05-18 01:48:04,084 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,084 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true
2016-05-18 01:48:04 Running Dag: dag_1463536083926_0001_1
2016-05-18 01:48:04,100 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536083926_0001_1, submitTime=1463536084073
2016-05-18 01:48:04,102 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001/tez-conf.pb" } size: 40937 timestamp: 1463536083000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:04,102 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,102 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:04,103 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001/tez-conf.pb" } size: 40937 timestamp: 1463536083000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:04,103 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,103 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:04,104 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:04,104 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536083926_0001_1, initTime=1463536084104
2016-05-18 01:48:04,104 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083926_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:04,104 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:04,105 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:DAG_STARTED]: dagID=dag_1463536083926_0001_1, startTime=1463536084105
2016-05-18 01:48:04,105 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083926_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:04,105 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B493BECD913C496EA7E00033917C5BE8 : {9858BD2AB64A4061B65A99716E26E6B6={InputName=9858BD2AB64A4061B65A99716E26E6B6}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:04,105 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 9858BD2AB64A4061B65A99716E26E6B6, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:04,105 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,105 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,106 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,106 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,107 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:04,107 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,108 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/true_false_true_true, applicationId=application_1463536083926_0001, dagName=[BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true
2016-05-18 01:48:04,109 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 9858BD2AB64A4061B65A99716E26E6B6 on vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,112 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 9858BD2AB64A4061B65A99716E26E6B6 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:04,112 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:04,113 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,113 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536083926_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:04,115 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3], numAdditionalOutputs=1
2016-05-18 01:48:04,115 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6F1EC064E58642059DF7C35A30146BF7, vertex=vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:04,117 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AAB17C9215D145A084FD1B3E83CFA0A3:6F1EC064E58642059DF7C35A30146BF7 using oldmapred API
2016-05-18 01:48:04,118 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6F1EC064E58642059DF7C35A30146BF7, vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:04,120 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, vertexId=vertex_1463536083926_0001_1_01, initRequestedTime=1463536084107, initedTime=1463536084115, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:04,121 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:04,121 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:04,122 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:04,122 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:04,122 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:04,123 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 9858BD2AB64A4061B65A99716E26E6B6 on vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,124 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:04,125 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {9858BD2AB64A4061B65A99716E26E6B6=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:04,127 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] parallelism set to 1
2016-05-18 01:48:04,128 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:04,128 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B493BECD913C496EA7E00033917C5BE8, vertexId=vertex_1463536083926_0001_1_00, initRequestedTime=1463536084105, initedTime=1463536084128, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:04,128 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] with distanceFromRoot: 0
2016-05-18 01:48:04,128 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:04,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536083926_0001_1_00, startRequestedTime=1463536084107, startedTime=1463536084128
2016-05-18 01:48:04,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536083926_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:04,129 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:04,129 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536083926_0001_1_00 for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] numStartedSources: 1 numSources: 1
2016-05-18 01:48:04,129 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] with distanceFromRoot: 1
2016-05-18 01:48:04,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536083926_0001_1_01, startRequestedTime=1463536084129, startedTime=1463536084129
2016-05-18 01:48:04,132 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:04,135 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AAB17C9215D145A084FD1B3E83CFA0A3 with 0 source tasks and 1 pending tasks
2016-05-18 01:48:04,136 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B493BECD913C496EA7E00033917C5BE8
2016-05-18 01:48:04,136 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,136 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B493BECD913C496EA7E00033917C5BE8 in vertex: AAB17C9215D145A084FD1B3E83CFA0A3 numBipartiteSourceTasks: 1
2016-05-18 01:48:04,138 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_STARTED]: vertexName=B493BECD913C496EA7E00033917C5BE8, taskId=task_1463536083926_0001_1_00_000000, scheduledTime=1463536084138, launchTime=1463536084138
2016-05-18 01:48:04,138 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,140 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:04,142 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536083926_0001_00_000001, launchTime=1463536084141
2016-05-18 01:48:04,142 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083926_0001_00_000001
2016-05-18 01:48:04,143 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083926_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,144 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536083926_0001_1_00_000000_0] started. Is using containerId: [container_1463536083926_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:04,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B493BECD913C496EA7E00033917C5BE8, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0, startTime=1463536084143, containerId=container_1463536083926_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:04 Starting to run new task attempt: attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,144 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,145 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,145 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:04,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true, VertexName: B493BECD913C496EA7E00033917C5BE8, VertexParallelism: 1, TaskAttemptID:attempt_1463536083926_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9858BD2AB64A4061B65A99716E26E6B6, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AAB17C9215D145A084FD1B3E83CFA0A3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:04,145 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:04,146 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536083926_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:04,147 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536083926_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:04,147 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,151 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:04,152 INFO input.MRInput (MRInput.java:initialize(440)) - 9858BD2AB64A4061B65A99716E26E6B6 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:04,152 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 9858BD2AB64A4061B65A99716E26E6B6
2016-05-18 01:48:04,152 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536083926_0001, with dag id: dag_1463536083926_0001_1
2016-05-18 01:48:04,155 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B493BECD913C496EA7E00033917C5BE8, ordinal: 0
2016-05-18 01:48:04,155 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B493BECD913C496EA7E00033917C5BE8, mem on start (mb), free: 240, total: 534, max: 672
2016-05-18 01:48:04,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:04,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:04,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:04,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:04,155 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[9858BD2AB64A4061B65A99716E26E6B6:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AAB17C9215D145A084FD1B3E83CFA0A3:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:04,155 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:04,156 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:04,156 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:04,156 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[9858BD2AB64A4061B65A99716E26E6B6:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AAB17C9215D145A084FD1B3E83CFA0A3:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:04,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:04,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 9858BD2AB64A4061B65A99716E26E6B6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:04,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 9858BD2AB64A4061B65A99716E26E6B6
2016-05-18 01:48:04,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:04,157 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:04,157 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:04,157 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,161 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:04,162 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 9858BD2AB64A4061B65A99716E26E6B6 initialized RecordReader from event
2016-05-18 01:48:04,164 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:04,165 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 9858BD2AB64A4061B65A99716E26E6B6
2016-05-18 01:48:04,165 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: 168B8D78F6D14B16A457F1164E189E1D
2016-05-18 01:48:04,165 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 168B8D78F6D14B16A457F1164E189E1D
2016-05-18 01:48:04,167 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AAB17C9215D145A084FD1B3E83CFA0A3 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@34762553, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@388c870d, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:04,167 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AAB17C9215D145A084FD1B3E83CFA0A3: , UsingHashComparator=false
2016-05-18 01:48:04,180 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:04,180 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:04,180 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AAB17C9215D145A084FD1B3E83CFA0A3: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:04,180 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AAB17C9215D145A084FD1B3E83CFA0A3: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:04,181 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:04,181 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B493BECD913C496EA7E00033917C5BE8, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:04,186 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,186 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B493BECD913C496EA7E00033917C5BE8, mem on close (mb), free: 138, total: 534, max: 672
2016-05-18 01:48:04,186 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AAB17C9215D145A084FD1B3E83CFA0A3: Starting flush of map output
2016-05-18 01:48:04,186 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AAB17C9215D145A084FD1B3E83CFA0A3: Span0.length = 37, perItem = 24
2016-05-18 01:48:04,188 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AAB17C9215D145A084FD1B3E83CFA0A3: done sorting span=0, length=37, time=1
2016-05-18 01:48:04,188 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AAB17C9215D145A084FD1B3E83CFA0A3: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:04,189 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AAB17C9215D145A084FD1B3E83CFA0A3: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-local-dir/output/attempt_1463536083926_0001_1_00_000000_0_10149_0/file.out
2016-05-18 01:48:04,192 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:04 Completed running task attempt: attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,192 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:04,193 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536083926_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:04,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536083926_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536084181, Process_Duration=5, Process_End_Time=1463536084186, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:04,194 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536083926_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,194 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B493BECD913C496EA7E00033917C5BE8, taskAttemptId=attempt_1463536083926_0001_1_00_000000_0, creationTime=1463536084138, allocationTime=1463536084141, startTime=1463536084143, finishTime=1463536084194, timeTaken=51, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:04,194 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536083926_0001_1_00_000000_0
2016-05-18 01:48:04,194 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,195 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:04,195 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_FINISHED]: vertexName=B493BECD913C496EA7E00033917C5BE8, taskId=task_1463536083926_0001_1_00_000000, startTime=1463536084143, finishTime=1463536084195, timeTaken=52, status=SUCCEEDED, successfulAttemptID=attempt_1463536083926_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536084181, Process_Duration=5, Process_End_Time=1463536084186, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:04,195 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,195 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083926_0001_00_000001
2016-05-18 01:48:04,195 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:04,196 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_FINISHED]: vertexName=B493BECD913C496EA7E00033917C5BE8, vertexId=vertex_1463536083926_0001_1_00, initRequestedTime=1463536084105, initedTime=1463536084128, startRequestedTime=1463536084107, startedTime=1463536084128, finishTime=1463536084196, timeTaken=68, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536084181, Process_Duration=5, Process_End_Time=1463536084186, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536084143, firstTasksToStart=[ task_1463536083926_0001_1_00_000000 ], lastTaskFinishTime=1463536084194, lastTasksToFinish=[ task_1463536083926_0001_1_00_000000 ], minTaskDuration=51, maxTaskDuration=51, avgTaskDuration=51.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536083926_0001_1_00_000000 ], longestDurationTasks=[ task_1463536083926_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:04,197 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:04,197 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] attempt: attempt_1463536083926_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:04,197 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AAB17C9215D145A084FD1B3E83CFA0A3
2016-05-18 01:48:04,197 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B493BECD913C496EA7E00033917C5BE8, destinationVertexName=AAB17C9215D145A084FD1B3E83CFA0A3. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:04,198 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536083926_0001_00_000001, stoppedTime=1463536084197, exitStatus=0
2016-05-18 01:48:04,198 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:04,198 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,198 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_STARTED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, taskId=task_1463536083926_0001_1_01_000000, scheduledTime=1463536084198, launchTime=1463536084198
2016-05-18 01:48:04,198 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536083926_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:04,198 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,201 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536083926_0001_00_000002, launchTime=1463536084201
2016-05-18 01:48:04,202 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083926_0001_00_000002
2016-05-18 01:48:04,202 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083926_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,202 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536083926_0001_1_01_000000_0] started. Is using containerId: [container_1463536083926_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:04 Starting to run new task attempt: attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0, startTime=1463536084202, containerId=container_1463536083926_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:04,203 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,203 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,203 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:04,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true, VertexName: AAB17C9215D145A084FD1B3E83CFA0A3, VertexParallelism: 1, TaskAttemptID:attempt_1463536083926_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B493BECD913C496EA7E00033917C5BE8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6F1EC064E58642059DF7C35A30146BF7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:04,204 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:04,205 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536083926_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,205 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536083926_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:04,205 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,206 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536083926_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:04,206 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083926_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:04,207 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536083926_0001_00_000001, Exiting
2016-05-18 01:48:04,207 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536083926_0001_00_000001
2016-05-18 01:48:04,207 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536083926_0001_00_000001
2016-05-18 01:48:04,209 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:04,211 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:04,212 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536083926_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:04,213 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536083926_0001_00_000001 completed successfully
2016-05-18 01:48:04,217 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AAB17C9215D145A084FD1B3E83CFA0A3, ordinal: 1
2016-05-18 01:48:04,217 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AAB17C9215D145A084FD1B3E83CFA0A3, mem on start (mb), free: 132, total: 534, max: 672
2016-05-18 01:48:04,217 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:04,217 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:04,218 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:04,221 INFO output.MROutput (MROutput.java:initialize(421)) - 6F1EC064E58642059DF7C35A30146BF7: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:04,221 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:04,221 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6F1EC064E58642059DF7C35A30146BF7:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B493BECD913C496EA7E00033917C5BE8:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:48:04,222 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:04,222 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:04,222 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:04,222 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6F1EC064E58642059DF7C35A30146BF7:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B493BECD913C496EA7E00033917C5BE8:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:48:04,222 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:04,222 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B493BECD913C496EA7E00033917C5BE8 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:04,223 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B493BECD913C496EA7E00033917C5BE8: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:04,223 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B493BECD913C496EA7E00033917C5BE8: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:04,224 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B493BECD913C496EA7E00033917C5BE8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:04,224 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B493BECD913C496EA7E00033917C5BE8
2016-05-18 01:48:04,226 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:04,226 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:04,226 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:04,226 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,230 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B493BECD913C496EA7E00033917C5BE8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:04,230 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:04,231 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: 168B8D78F6D14B16A457F1164E189E1D
2016-05-18 01:48:04,231 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true"], id: 6F1EC064E58642059DF7C35A30146BF7
2016-05-18 01:48:04,232 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true"]
2016-05-18 01:48:04,232 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] 168B8D78F6D14B16A457F1164E189E1D, for 1 inputs
2016-05-18 01:48:04,233 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536083926_0001_1_00_000000_0_10149} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536084232, TimeTaken=2, Rate=0.35 MB/s
2016-05-18 01:48:04,233 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B493BECD913C496EA7E00033917C5BE8
2016-05-18 01:48:04,233 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:04,233 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B493BECD913C496EA7E00033917C5BE8, wasInterrupted=false
2016-05-18 01:48:04,234 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:04,235 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:04,236 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001_wd/localmode-local-dir/output/attempt_1463536083926_0001_1_00_000000_0_10149/file.out, len=732, isLocal=true
2016-05-18 01:48:04,236 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:04,236 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:04,236 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:04,236 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:04,237 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B493BECD913C496EA7E00033917C5BE8
2016-05-18 01:48:04,237 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B493BECD913C496EA7E00033917C5BE8: Shuffle Runner thread complete
2016-05-18 01:48:04,237 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AAB17C9215D145A084FD1B3E83CFA0A3, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:48:04,238 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B493BECD913C496EA7E00033917C5BE8: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:04,242 INFO output.MROutput (MROutput.java:close(526)) - 6F1EC064E58642059DF7C35A30146BF7 closed
2016-05-18 01:48:04,243 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536083926_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:04,243 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,243 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AAB17C9215D145A084FD1B3E83CFA0A3, mem on close (mb), free: 128, total: 534, max: 672
2016-05-18 01:48:04,243 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B493BECD913C496EA7E00033917C5BE8
2016-05-18 01:48:04,243 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B493BECD913C496EA7E00033917C5BE8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:04,244 INFO output.MROutput (MROutput.java:close(526)) - 6F1EC064E58642059DF7C35A30146BF7 closed
2016-05-18 01:48:04,244 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:04 Completed running task attempt: attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,244 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:04,244 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536083926_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:04,245 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536083926_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536084237, Process_Duration=6, Process_End_Time=1463536084243, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:04,245 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536083926_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,245 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,245 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:04,245 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,245 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, taskAttemptId=attempt_1463536083926_0001_1_01_000000_0, creationTime=1463536084198, allocationTime=1463536084200, startTime=1463536084202, finishTime=1463536084245, timeTaken=43, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:04,245 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536083926_0001_00_000002
2016-05-18 01:48:04,246 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536083926_0001_1_01_000000_0
2016-05-18 01:48:04,246 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:TASK_FINISHED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, taskId=task_1463536083926_0001_1_01_000000, startTime=1463536084202, finishTime=1463536084246, timeTaken=44, status=SUCCEEDED, successfulAttemptID=attempt_1463536083926_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536084237, Process_Duration=6, Process_End_Time=1463536084243, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:04,246 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:04,246 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3]
2016-05-18 01:48:04,247 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:VERTEX_FINISHED]: vertexName=AAB17C9215D145A084FD1B3E83CFA0A3, vertexId=vertex_1463536083926_0001_1_01, initRequestedTime=1463536084107, initedTime=1463536084115, startRequestedTime=1463536084129, startedTime=1463536084129, finishTime=1463536084247, timeTaken=118, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536084237, Process_Duration=6, Process_End_Time=1463536084243, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536084202, firstTasksToStart=[ task_1463536083926_0001_1_01_000000 ], lastTaskFinishTime=1463536084245, lastTasksToFinish=[ task_1463536083926_0001_1_01_000000 ], minTaskDuration=43, maxTaskDuration=43, avgTaskDuration=43.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536083926_0001_1_01_000000 ], longestDurationTasks=[ task_1463536083926_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:04,247 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:04,248 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536083926_0001_00_000002, stoppedTime=1463536084248, exitStatus=0
2016-05-18 01:48:04,248 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:04,248 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,248 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536083926_0001_1_00 [B493BECD913C496EA7E00033917C5BE8]
2016-05-18 01:48:04,248 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536083926_0001_1
2016-05-18 01:48:04,248 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536083926_0001_1
2016-05-18 01:48:04,249 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083926_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:04,249 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6F1EC064E58642059DF7C35A30146BF7 for vertex: vertex_1463536083926_0001_1_01 [AAB17C9215D145A084FD1B3E83CFA0A3], outputName: 6F1EC064E58642059DF7C35A30146BF7
2016-05-18 01:48:04,250 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536083926_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:04,256 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536083926_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:04,256 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536083926_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:04,256 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536083926_0001_00_000002, Exiting
2016-05-18 01:48:04,256 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536083926_0001_00_000002
2016-05-18 01:48:04,256 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536083926_0001_00_000002
2016-05-18 01:48:04,256 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536083926_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:04,256 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536083926_0001_00_000002 completed successfully
2016-05-18 01:48:04,261 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6F1EC064E58642059DF7C35A30146BF7 of vertex/vertexGroup:AAB17C9215D145A084FD1B3E83CFA0A3 isVertexGroupOutput:false
2016-05-18 01:48:04,261 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,263 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536083926_0001_1, queueSize=0
2016-05-18 01:48:04,263 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536083926_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536083926_0001_1, startTime=1463536084104, finishTime=1463536084261, timeTaken=157, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072168418, Process_Duration=11, Process_End_Time=2927072168429, Read_Duration=4, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:04,263 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536083926_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:04,263 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536083926_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:04 Completed Dag: dag_1463536083926_0001_1
2016-05-18 01:48:04,264 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536083926_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:04,264 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536083926_0001_1_post
2016-05-18 01:48:04,264 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,264 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:04,265 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:04,265 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true, with id=dag_1463536083926_0001_1
2016-05-18 01:48:04,265 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27] comprehensivesort/(1/1) ...sive/true_false_true_true, with id=dag_1463536083926_0001_1
2016-05-18 01:48:04,270 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/true_false_true_true, applicationId=application_1463536083926_0001
2016-05-18 01:48:04,271 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:04,271 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:04,271 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:04,271 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:04,271 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:04,272 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:04,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:04,272 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:04,273 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:04,273 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:04,273 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:04,273 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:04,274 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:04,275 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:04,278 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD77A4AF2B144FF6AA1D506B5EC20038/F2EDF35DA98F4567B84F3E3EBC42AA27/.tez/application_1463536083926_0001
2016-05-18 01:48:04,279 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true/_temporary
2016-05-18 01:48:04,279 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.373
2016-05-18 01:48:04,377 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:04,378 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:04,413 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 01:48:04,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 01:48:04,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:04,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:04,420 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:04,434 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:04,442 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true
2016-05-18 01:48:04,502 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:04,502 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:04,502 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true"]
2016-05-18 01:48:04,503 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:04,503 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:04,503 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:04,503 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...nsive/true_true_true_true
2016-05-18 01:48:04,504 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:04,516 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:04,516 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:04,516 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:04,521 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:04,521 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:04,521 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:04,530 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001 doesn't exist and is created
2016-05-18 01:48:04,549 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:04,549 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:04,550 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd
2016-05-18 01:48:04,565 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536084521_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:04,565 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:04,565 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:04,567 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:04,567 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:04,567 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:04,567 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:04,568 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:04,570 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:04,570 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:04,570 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:04,570 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:04,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:04,571 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:04,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:04,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:04,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:04,571 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:04,572 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:04,572 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:04,575 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:04,575 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:04,576 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:04,577 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-log-dir
2016-05-18 01:48:04,577 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-log-dir/history.txt.appattempt_1463536084521_0001_000000, maxErrors=10
2016-05-18 01:48:04,577 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:04,577 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536084521_0001_000000, appSubmitTime=1463536084564, launchTime=1463536084564
2016-05-18 01:48:04,580 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:04,584 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56297
2016-05-18 01:48:04,584 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536084521
2016-05-18 01:48:04,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536084521_0001_000000, startTime=1463536084586
2016-05-18 01:48:04,586 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:04,650 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:04,650 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:04,651 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...nsive/true_true_true_true, applicationId=application_1463536084521_0001, dagName=[38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true
2016-05-18 01:48:04,666 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true
2016-05-18 01:48:04,666 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:04,667 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536084521_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-log-dir/dag_1463536084521_0001_1.dot
2016-05-18 01:48:04,667 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-log-dir/dag_1463536084521_0001_1-tez-dag.pb.txt
2016-05-18 01:48:04,673 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536084521_0001_1
2016-05-18 01:48:04,673 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,673 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true
2016-05-18 01:48:04 Running Dag: dag_1463536084521_0001_1
2016-05-18 01:48:04,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536084521_0001_1, submitTime=1463536084666
2016-05-18 01:48:04,689 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001/tez-conf.pb" } size: 40937 timestamp: 1463536084000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:04,689 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,689 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:04,690 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001/tez-conf.pb" } size: 40937 timestamp: 1463536084000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:04,690 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,690 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:04,690 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:04,691 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536084521_0001_1, initTime=1463536084691
2016-05-18 01:48:04,691 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536084521_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:04,691 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:04,691 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:DAG_STARTED]: dagID=dag_1463536084521_0001_1, startTime=1463536084691
2016-05-18 01:48:04,691 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536084521_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:04,691 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A3C22241C5BC445ABC8A24079C4FD5F7 : {3DAC05579E584F87A1F72EF0C3DD218D={InputName=3DAC05579E584F87A1F72EF0C3DD218D}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:04,692 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 3DAC05579E584F87A1F72EF0C3DD218D, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:04,692 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,692 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,692 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,692 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,692 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...nsive/true_true_true_true, applicationId=application_1463536084521_0001, dagName=[38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true
2016-05-18 01:48:04,694 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:04,694 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,694 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 3DAC05579E584F87A1F72EF0C3DD218D on vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,696 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 3DAC05579E584F87A1F72EF0C3DD218D asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:04,697 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:04,697 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,697 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,698 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:04,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536084521_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:04,698 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900], numAdditionalOutputs=1
2016-05-18 01:48:04,698 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=B9E216834076495E99F9CD01D654BA2B, vertex=vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:04,698 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:04,700 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:04,700 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:04,700 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F6A2DF1959E44651BC4DB516F16A1900:B9E216834076495E99F9CD01D654BA2B using oldmapred API
2016-05-18 01:48:04,700 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 3DAC05579E584F87A1F72EF0C3DD218D on vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,700 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=B9E216834076495E99F9CD01D654BA2B, vertexName=F6A2DF1959E44651BC4DB516F16A1900, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:04,701 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, vertexId=vertex_1463536084521_0001_1_01, initRequestedTime=1463536084694, initedTime=1463536084698, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:04,702 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:04,702 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:04,702 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {3DAC05579E584F87A1F72EF0C3DD218D=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:04,703 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] parallelism set to 1
2016-05-18 01:48:04,703 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:04,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, vertexId=vertex_1463536084521_0001_1_00, initRequestedTime=1463536084691, initedTime=1463536084703, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:04,703 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] with distanceFromRoot: 0
2016-05-18 01:48:04,703 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:04,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536084521_0001_1_00, startRequestedTime=1463536084694, startedTime=1463536084703
2016-05-18 01:48:04,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536084521_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:04,704 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:04,704 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536084521_0001_1_00 for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] numStartedSources: 1 numSources: 1
2016-05-18 01:48:04,704 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] with distanceFromRoot: 1
2016-05-18 01:48:04,705 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536084521_0001_1_01, startRequestedTime=1463536084704, startedTime=1463536084704
2016-05-18 01:48:04,705 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:04,706 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F6A2DF1959E44651BC4DB516F16A1900 with 0 source tasks and 1 pending tasks
2016-05-18 01:48:04,707 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A3C22241C5BC445ABC8A24079C4FD5F7 in vertex: F6A2DF1959E44651BC4DB516F16A1900 numBipartiteSourceTasks: 1
2016-05-18 01:48:04,707 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A3C22241C5BC445ABC8A24079C4FD5F7
2016-05-18 01:48:04,707 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_STARTED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, taskId=task_1463536084521_0001_1_00_000000, scheduledTime=1463536084708, launchTime=1463536084708
2016-05-18 01:48:04,708 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,708 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:04,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536084521_0001_00_000001, launchTime=1463536084710
2016-05-18 01:48:04,710 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536084521_0001_00_000001
2016-05-18 01:48:04,711 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536084521_0001_1_00_000000_0] started. Is using containerId: [container_1463536084521_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:04,711 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0, startTime=1463536084710, containerId=container_1463536084521_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:04,711 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536084521_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04 Starting to run new task attempt: attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,712 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,712 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,712 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:04,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true, VertexName: A3C22241C5BC445ABC8A24079C4FD5F7, VertexParallelism: 1, TaskAttemptID:attempt_1463536084521_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3DAC05579E584F87A1F72EF0C3DD218D, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F6A2DF1959E44651BC4DB516F16A1900, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:04,712 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:04,715 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,715 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536084521_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:04,716 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536084521_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:04,716 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536084521_0001, with dag id: dag_1463536084521_0001_1
2016-05-18 01:48:04,720 INFO input.MRInput (MRInput.java:initialize(440)) - 3DAC05579E584F87A1F72EF0C3DD218D using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:04,720 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:04,720 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 3DAC05579E584F87A1F72EF0C3DD218D
2016-05-18 01:48:04,723 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A3C22241C5BC445ABC8A24079C4FD5F7, ordinal: 0
2016-05-18 01:48:04,724 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A3C22241C5BC445ABC8A24079C4FD5F7, mem on start (mb), free: 186, total: 535, max: 672
2016-05-18 01:48:04,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:04,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:04,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:04,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:04,724 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[3DAC05579E584F87A1F72EF0C3DD218D:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F6A2DF1959E44651BC4DB516F16A1900:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:04,724 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:04,725 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:04,725 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:04,725 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[3DAC05579E584F87A1F72EF0C3DD218D:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F6A2DF1959E44651BC4DB516F16A1900:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:04,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:04,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 3DAC05579E584F87A1F72EF0C3DD218D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:04,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:04,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:04,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 3DAC05579E584F87A1F72EF0C3DD218D
2016-05-18 01:48:04,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:04,727 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,729 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:04,729 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 3DAC05579E584F87A1F72EF0C3DD218D initialized RecordReader from event
2016-05-18 01:48:04,731 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:04,731 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 3DAC05579E584F87A1F72EF0C3DD218D
2016-05-18 01:48:04,732 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: D012C12DADFB450BB6E9E2EE54053E7B
2016-05-18 01:48:04,732 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] D012C12DADFB450BB6E9E2EE54053E7B
2016-05-18 01:48:04,734 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F6A2DF1959E44651BC4DB516F16A1900 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2c8b16b6, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@54158d39, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:04,735 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F6A2DF1959E44651BC4DB516F16A1900: , UsingHashComparator=false
2016-05-18 01:48:04,847 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:04,848 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:04,848 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F6A2DF1959E44651BC4DB516F16A1900: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:04,848 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F6A2DF1959E44651BC4DB516F16A1900: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:04,848 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:04,849 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A3C22241C5BC445ABC8A24079C4FD5F7, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:04,853 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,853 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A3C22241C5BC445ABC8A24079C4FD5F7, mem on close (mb), free: 184, total: 636, max: 672
2016-05-18 01:48:04,853 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F6A2DF1959E44651BC4DB516F16A1900: Starting flush of map output
2016-05-18 01:48:04,853 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F6A2DF1959E44651BC4DB516F16A1900: Span0.length = 37, perItem = 24
2016-05-18 01:48:04,854 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F6A2DF1959E44651BC4DB516F16A1900: done sorting span=0, length=37, time=1
2016-05-18 01:48:04,854 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F6A2DF1959E44651BC4DB516F16A1900: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:04,856 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F6A2DF1959E44651BC4DB516F16A1900: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-local-dir/output/attempt_1463536084521_0001_1_00_000000_0_10155_0/file.out
2016-05-18 01:48:04,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:04 Completed running task attempt: attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,860 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:04,860 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536084521_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:04,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536084521_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536084849, Process_Duration=4, Process_End_Time=1463536084853, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:04,861 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536084521_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,861 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,861 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:04,861 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,862 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536084521_0001_00_000001
2016-05-18 01:48:04,862 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, taskAttemptId=attempt_1463536084521_0001_1_00_000000_0, creationTime=1463536084708, allocationTime=1463536084709, startTime=1463536084710, finishTime=1463536084861, timeTaken=151, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:04,863 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536084521_0001_1_00_000000_0
2016-05-18 01:48:04,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_FINISHED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, taskId=task_1463536084521_0001_1_00_000000, startTime=1463536084710, finishTime=1463536084863, timeTaken=153, status=SUCCEEDED, successfulAttemptID=attempt_1463536084521_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536084849, Process_Duration=4, Process_End_Time=1463536084853, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:04,864 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:04,864 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,864 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_FINISHED]: vertexName=A3C22241C5BC445ABC8A24079C4FD5F7, vertexId=vertex_1463536084521_0001_1_00, initRequestedTime=1463536084691, initedTime=1463536084703, startRequestedTime=1463536084694, startedTime=1463536084703, finishTime=1463536084864, timeTaken=161, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536084849, Process_Duration=4, Process_End_Time=1463536084853, Read_Duration=1, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536084710, firstTasksToStart=[ task_1463536084521_0001_1_00_000000 ], lastTaskFinishTime=1463536084861, lastTasksToFinish=[ task_1463536084521_0001_1_00_000000 ], minTaskDuration=151, maxTaskDuration=151, avgTaskDuration=151.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536084521_0001_1_00_000000 ], longestDurationTasks=[ task_1463536084521_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:04,864 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:04,865 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] attempt: attempt_1463536084521_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:04,865 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:04,865 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,865 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F6A2DF1959E44651BC4DB516F16A1900
2016-05-18 01:48:04,865 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A3C22241C5BC445ABC8A24079C4FD5F7, destinationVertexName=F6A2DF1959E44651BC4DB516F16A1900. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:04,866 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,866 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_STARTED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, taskId=task_1463536084521_0001_1_01_000000, scheduledTime=1463536084866, launchTime=1463536084866
2016-05-18 01:48:04,866 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536084521_0001_00_000001, stoppedTime=1463536084866, exitStatus=0
2016-05-18 01:48:04,867 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,867 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536084521_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:04,869 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536084521_0001_00_000002, launchTime=1463536084869
2016-05-18 01:48:04,869 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536084521_0001_00_000002
2016-05-18 01:48:04,870 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536084521_0001_1_01_000000_0] started. Is using containerId: [container_1463536084521_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:04,870 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0, startTime=1463536084870, containerId=container_1463536084521_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:04,870 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536084521_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04 Starting to run new task attempt: attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,871 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,871 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,871 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:04,871 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true, VertexName: F6A2DF1959E44651BC4DB516F16A1900, VertexParallelism: 1, TaskAttemptID:attempt_1463536084521_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A3C22241C5BC445ABC8A24079C4FD5F7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B9E216834076495E99F9CD01D654BA2B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:04,871 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:04,872 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536084521_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,872 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536084521_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:04,872 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536084521_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:04,872 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536084521_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:04,873 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536084521_0001_00_000001, Exiting
2016-05-18 01:48:04,873 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536084521_0001_00_000001
2016-05-18 01:48:04,873 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,873 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536084521_0001_00_000001
2016-05-18 01:48:04,875 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:04,876 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536084521_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:04,876 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536084521_0001_00_000001 completed successfully
2016-05-18 01:48:04,879 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:04,884 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F6A2DF1959E44651BC4DB516F16A1900, ordinal: 1
2016-05-18 01:48:04,884 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F6A2DF1959E44651BC4DB516F16A1900, mem on start (mb), free: 179, total: 636, max: 672
2016-05-18 01:48:04,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:04,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:04,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:04,894 INFO output.MROutput (MROutput.java:initialize(421)) - B9E216834076495E99F9CD01D654BA2B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:04,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:04,894 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A3C22241C5BC445ABC8A24079C4FD5F7:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [B9E216834076495E99F9CD01D654BA2B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:48:04,894 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:04,894 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:04,895 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:04,895 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A3C22241C5BC445ABC8A24079C4FD5F7:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [B9E216834076495E99F9CD01D654BA2B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:48:04,895 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:04,895 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A3C22241C5BC445ABC8A24079C4FD5F7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:04,895 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A3C22241C5BC445ABC8A24079C4FD5F7: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:04,895 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A3C22241C5BC445ABC8A24079C4FD5F7: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:04,896 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A3C22241C5BC445ABC8A24079C4FD5F7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:04,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A3C22241C5BC445ABC8A24079C4FD5F7
2016-05-18 01:48:04,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:04,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:04,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:04,897 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,902 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A3C22241C5BC445ABC8A24079C4FD5F7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:04,903 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:04,904 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: D012C12DADFB450BB6E9E2EE54053E7B
2016-05-18 01:48:04,904 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true"], id: B9E216834076495E99F9CD01D654BA2B
2016-05-18 01:48:04,904 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true"]
2016-05-18 01:48:04,905 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] D012C12DADFB450BB6E9E2EE54053E7B, for 1 inputs
2016-05-18 01:48:04,905 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536084521_0001_1_00_000000_0_10155} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536084905, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:48:04,905 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A3C22241C5BC445ABC8A24079C4FD5F7
2016-05-18 01:48:04,905 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:04,905 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A3C22241C5BC445ABC8A24079C4FD5F7, wasInterrupted=false
2016-05-18 01:48:04,907 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:04,907 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A3C22241C5BC445ABC8A24079C4FD5F7: Already shutdown. Ignoring fetch complete
2016-05-18 01:48:04,908 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:04,909 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001_wd/localmode-local-dir/output/attempt_1463536084521_0001_1_00_000000_0_10155/file.out, len=732, isLocal=true
2016-05-18 01:48:04,909 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:04,909 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:04,909 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:04,910 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:04,910 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A3C22241C5BC445ABC8A24079C4FD5F7
2016-05-18 01:48:04,910 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A3C22241C5BC445ABC8A24079C4FD5F7: Shuffle Runner thread complete
2016-05-18 01:48:04,910 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F6A2DF1959E44651BC4DB516F16A1900, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:48:04,911 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A3C22241C5BC445ABC8A24079C4FD5F7: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:04,915 INFO output.MROutput (MROutput.java:close(526)) - B9E216834076495E99F9CD01D654BA2B closed
2016-05-18 01:48:04,915 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536084521_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:04,916 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,916 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F6A2DF1959E44651BC4DB516F16A1900, mem on close (mb), free: 174, total: 636, max: 672
2016-05-18 01:48:04,916 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A3C22241C5BC445ABC8A24079C4FD5F7
2016-05-18 01:48:04,917 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A3C22241C5BC445ABC8A24079C4FD5F7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:04,917 INFO output.MROutput (MROutput.java:close(526)) - B9E216834076495E99F9CD01D654BA2B closed
2016-05-18 01:48:04,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:04 Completed running task attempt: attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:04,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536084521_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:04,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536084521_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536084910, Process_Duration=6, Process_End_Time=1463536084916, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:04,919 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536084521_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, taskAttemptId=attempt_1463536084521_0001_1_01_000000_0, creationTime=1463536084866, allocationTime=1463536084867, startTime=1463536084870, finishTime=1463536084919, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:04,920 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:04,920 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:04,920 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,920 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536084521_0001_1_01_000000_0
2016-05-18 01:48:04,921 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536084521_0001_00_000002
2016-05-18 01:48:04,921 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:TASK_FINISHED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, taskId=task_1463536084521_0001_1_01_000000, startTime=1463536084870, finishTime=1463536084921, timeTaken=51, status=SUCCEEDED, successfulAttemptID=attempt_1463536084521_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536084910, Process_Duration=6, Process_End_Time=1463536084916, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:04,922 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:04,922 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900]
2016-05-18 01:48:04,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:VERTEX_FINISHED]: vertexName=F6A2DF1959E44651BC4DB516F16A1900, vertexId=vertex_1463536084521_0001_1_01, initRequestedTime=1463536084694, initedTime=1463536084698, startRequestedTime=1463536084704, startedTime=1463536084704, finishTime=1463536084922, timeTaken=218, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536084910, Process_Duration=6, Process_End_Time=1463536084916, Read_Duration=0, Tuples_Read=37, Tuples_Written=37, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536084870, firstTasksToStart=[ task_1463536084521_0001_1_01_000000 ], lastTaskFinishTime=1463536084919, lastTasksToFinish=[ task_1463536084521_0001_1_01_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536084521_0001_1_01_000000 ], longestDurationTasks=[ task_1463536084521_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:04,923 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:04,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536084521_0001_00_000002, stoppedTime=1463536084923, exitStatus=0
2016-05-18 01:48:04,923 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:04,923 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,923 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536084521_0001_1_00 [A3C22241C5BC445ABC8A24079C4FD5F7]
2016-05-18 01:48:04,923 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536084521_0001_1
2016-05-18 01:48:04,924 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536084521_0001_1
2016-05-18 01:48:04,924 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536084521_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:04,924 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: B9E216834076495E99F9CD01D654BA2B for vertex: vertex_1463536084521_0001_1_01 [F6A2DF1959E44651BC4DB516F16A1900], outputName: B9E216834076495E99F9CD01D654BA2B
2016-05-18 01:48:04,924 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536084521_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:04,925 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536084521_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:04,925 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536084521_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:04,925 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536084521_0001_00_000002, Exiting
2016-05-18 01:48:04,925 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536084521_0001_00_000002
2016-05-18 01:48:04,926 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536084521_0001_00_000002
2016-05-18 01:48:04,926 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536084521_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:04,926 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536084521_0001_00_000002 completed successfully
2016-05-18 01:48:04,934 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:B9E216834076495E99F9CD01D654BA2B of vertex/vertexGroup:F6A2DF1959E44651BC4DB516F16A1900 isVertexGroupOutput:false
2016-05-18 01:48:04,934 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:04,936 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536084521_0001_1, queueSize=0
2016-05-18 01:48:04,937 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536084521_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536084521_0001_1, startTime=1463536084691, finishTime=1463536084934, timeTaken=243, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=380, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072169759, Process_Duration=10, Process_End_Time=2927072169769, Read_Duration=1, Tuples_Read=74, Tuples_Written=74, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:04,937 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536084521_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:04,937 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536084521_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:04 Completed Dag: dag_1463536084521_0001_1
2016-05-18 01:48:04,937 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536084521_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:04,937 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536084521_0001_1_post
2016-05-18 01:48:04,937 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:04,937 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:04,937 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:04,938 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true, with id=dag_1463536084521_0001_1
2016-05-18 01:48:04,938 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429] comprehensivesort/(1/1) ...nsive/true_true_true_true, with id=dag_1463536084521_0001_1
2016-05-18 01:48:04,944 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...nsive/true_true_true_true, applicationId=application_1463536084521_0001
2016-05-18 01:48:04,945 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:04,945 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:04,945 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:04,945 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:04,945 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:04,945 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:04,946 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:04,946 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:04,946 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:04,946 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:04,946 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:04,946 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:04,946 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:04,947 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:04,950 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/38C54007185A495ABD55E9928011BFA1/E2461C11AEA3454185367748B8E5E429/.tez/application_1463536084521_0001
2016-05-18 01:48:04,950 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true/_temporary
2016-05-18 01:48:04,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.448
2016-05-18 01:48:05,019 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:48:05,019 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:48:05,068 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 01:48:05,074 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 01:48:05,075 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:05,075 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:48:05,075 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:48:05,089 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt
2016-05-18 01:48:05,095 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true
2016-05-18 01:48:05,112 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:48:05,113 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:48:05,146 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting
2016-05-18 01:48:05,146 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:05,146 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] sink: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true"]
2016-05-18 01:48:05,146 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] parallel execution of steps is enabled: false
2016-05-18 01:48:05,147 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] executing total steps: 1
2016-05-18 01:48:05,147 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] allocating management threads: 1
2016-05-18 01:48:05,148 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] starting step: (1/1) ...sive/false_true_true_true
2016-05-18 01:48:05,149 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [comprehensivesort] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:48:05,185 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] tez session mode enabled: true
2016-05-18 01:48:05,190 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:05,190 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:48:05,190 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:48:05,200 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001 doesn't exist and is created
2016-05-18 01:48:05,221 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:48:05,221 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:48:05,222 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd
2016-05-18 01:48:05,238 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536085190_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:48:05,239 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:05,239 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:48:05,240 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:48:05,240 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:48:05,240 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:48:05,240 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:48:05,241 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:48:05,242 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:48:05,242 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:05,243 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:48:05,243 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:48:05,243 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:48:05,244 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:48:05,245 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:48:05,245 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:48:05,245 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-log-dir
2016-05-18 01:48:05,245 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-log-dir/history.txt.appattempt_1463536085190_0001_000000, maxErrors=10
2016-05-18 01:48:05,246 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:48:05,246 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536085190_0001_000000, appSubmitTime=1463536085238, launchTime=1463536085238
2016-05-18 01:48:05,248 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:48:05,255 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:58704
2016-05-18 01:48:05,256 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536085190
2016-05-18 01:48:05,258 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536085190_0001_000000, startTime=1463536085258
2016-05-18 01:48:05,258 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:48:05,321 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:48:05,321 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:48:05,322 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...sive/false_true_true_true, applicationId=application_1463536085190_0001, dagName=[899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true
2016-05-18 01:48:05,338 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true
2016-05-18 01:48:05,338 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:48:05,339 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536085190_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-log-dir/dag_1463536085190_0001_1.dot
2016-05-18 01:48:05,340 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-log-dir/dag_1463536085190_0001_1-tez-dag.pb.txt
2016-05-18 01:48:05,347 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536085190_0001_1
2016-05-18 01:48:05,348 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,348 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true
2016-05-18 01:48:05 Running Dag: dag_1463536085190_0001_1
2016-05-18 01:48:05,365 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536085190_0001_1, submitTime=1463536085338
2016-05-18 01:48:05,366 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001/tez-conf.pb" } size: 40937 timestamp: 1463536085000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:05,366 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,367 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:05,368 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001/tez-conf.pb" } size: 40937 timestamp: 1463536085000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:48:05,368 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,368 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:48:05,368 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:48:05,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536085190_0001_1, initTime=1463536085369
2016-05-18 01:48:05,369 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536085190_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:48:05,369 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:48:05,370 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:DAG_STARTED]: dagID=dag_1463536085190_0001_1, startTime=1463536085370
2016-05-18 01:48:05,370 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536085190_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:48:05,370 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...sive/false_true_true_true, applicationId=application_1463536085190_0001, dagName=[899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true
2016-05-18 01:48:05,370 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B8B4DC5415724A5DB95209889E072D11 : {2D6B230F2E9241E18DBD1B076BF23F42={InputName=2D6B230F2E9241E18DBD1B076BF23F42}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:48:05,370 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 2D6B230F2E9241E18DBD1B076BF23F42, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:48:05,370 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,370 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,371 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,371 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,377 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:48:05,377 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,378 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 2D6B230F2E9241E18DBD1B076BF23F42 on vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,380 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:48:05,380 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,380 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,381 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536085190_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:48:05,381 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86], numAdditionalOutputs=1
2016-05-18 01:48:05,381 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=9C2B51A1E69045068C2451F593151C37, vertex=vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:48:05,380 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 2D6B230F2E9241E18DBD1B076BF23F42 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:48:05,383 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E1E9A926FB8D4B7782C881430107EC86:9C2B51A1E69045068C2451F593151C37 using oldmapred API
2016-05-18 01:48:05,383 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=9C2B51A1E69045068C2451F593151C37, vertexName=E1E9A926FB8D4B7782C881430107EC86, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:48:05,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, vertexId=vertex_1463536085190_0001_1_01, initRequestedTime=1463536085377, initedTime=1463536085381, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:48:05,384 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:48:05,385 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:48:05,385 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 221 Min splitLength: 52428800 New desired splits: 1 Total length: 221 Original splits: 2
2016-05-18 01:48:05,385 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 221 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 221 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:48:05,385 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:48:05,387 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 2D6B230F2E9241E18DBD1B076BF23F42 on vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,387 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:05,388 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {2D6B230F2E9241E18DBD1B076BF23F42=forAllWorkUnits=true, update=[1]}
2016-05-18 01:48:05,388 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] parallelism set to 1
2016-05-18 01:48:05,388 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:48:05,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B8B4DC5415724A5DB95209889E072D11, vertexId=vertex_1463536085190_0001_1_00, initRequestedTime=1463536085370, initedTime=1463536085389, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:48:05,389 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] with distanceFromRoot: 0
2016-05-18 01:48:05,389 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:48:05,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536085190_0001_1_00, startRequestedTime=1463536085377, startedTime=1463536085389
2016-05-18 01:48:05,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536085190_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:48:05,390 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:05,390 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536085190_0001_1_00 for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] numStartedSources: 1 numSources: 1
2016-05-18 01:48:05,390 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] with distanceFromRoot: 1
2016-05-18 01:48:05,390 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536085190_0001_1_01, startRequestedTime=1463536085390, startedTime=1463536085390
2016-05-18 01:48:05,391 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:48:05,391 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E1E9A926FB8D4B7782C881430107EC86 with 0 source tasks and 1 pending tasks
2016-05-18 01:48:05,394 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B8B4DC5415724A5DB95209889E072D11
2016-05-18 01:48:05,394 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B8B4DC5415724A5DB95209889E072D11 in vertex: E1E9A926FB8D4B7782C881430107EC86 numBipartiteSourceTasks: 1
2016-05-18 01:48:05,395 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,395 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_STARTED]: vertexName=B8B4DC5415724A5DB95209889E072D11, taskId=task_1463536085190_0001_1_00_000000, scheduledTime=1463536085395, launchTime=1463536085395
2016-05-18 01:48:05,395 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,396 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:48:05,398 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536085190_0001_00_000001, launchTime=1463536085398
2016-05-18 01:48:05,398 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] submitted tez dag to app master: application_1463536085190_0001, with dag id: dag_1463536085190_0001_1
2016-05-18 01:48:05,398 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536085190_0001_00_000001
2016-05-18 01:48:05,399 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536085190_0001_1_00_000000_0] started. Is using containerId: [container_1463536085190_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:48:05,399 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B8B4DC5415724A5DB95209889E072D11, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0, startTime=1463536085399, containerId=container_1463536085190_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:48:05,400 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536085190_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05 Starting to run new task attempt: attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,400 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,401 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,401 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:05,401 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true, VertexName: B8B4DC5415724A5DB95209889E072D11, VertexParallelism: 1, TaskAttemptID:attempt_1463536085190_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=2D6B230F2E9241E18DBD1B076BF23F42, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E1E9A926FB8D4B7782C881430107EC86, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:48:05,401 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:05,402 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,402 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536085190_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:48:05,402 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536085190_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:48:05,404 INFO input.MRInput (MRInput.java:initialize(440)) - 2D6B230F2E9241E18DBD1B076BF23F42 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:48:05,408 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 2D6B230F2E9241E18DBD1B076BF23F42
2016-05-18 01:48:05,409 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:05,414 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B8B4DC5415724A5DB95209889E072D11, ordinal: 0
2016-05-18 01:48:05,414 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8B4DC5415724A5DB95209889E072D11, mem on start (mb), free: 228, total: 634, max: 672
2016-05-18 01:48:05,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:05,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:05,415 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:05,415 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:05,415 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[2D6B230F2E9241E18DBD1B076BF23F42:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E1E9A926FB8D4B7782C881430107EC86:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:48:05,415 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:05,415 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:05,415 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:48:05,415 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[2D6B230F2E9241E18DBD1B076BF23F42:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E1E9A926FB8D4B7782C881430107EC86:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:48:05,415 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:05,415 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 2D6B230F2E9241E18DBD1B076BF23F42 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:05,416 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:05,416 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:05,416 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 2D6B230F2E9241E18DBD1B076BF23F42
2016-05-18 01:48:05,416 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:05,416 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,419 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:05,419 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"] streamed: true, id: 2D6B230F2E9241E18DBD1B076BF23F42
2016-05-18 01:48:05,420 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(comprehensivesort)[by:[{1}:'num']], id: A8B216B562144D48BCE5378399D2A7EF
2016-05-18 01:48:05,420 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A8B216B562144D48BCE5378399D2A7EF
2016-05-18 01:48:05,420 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:0+110, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt:110+111], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=221}
2016-05-18 01:48:05,420 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 2D6B230F2E9241E18DBD1B076BF23F42 initialized RecordReader from event
2016-05-18 01:48:05,421 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E1E9A926FB8D4B7782C881430107EC86 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@795654ef, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@12e25525, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:48:05,422 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E1E9A926FB8D4B7782C881430107EC86: , UsingHashComparator=false
2016-05-18 01:48:05,434 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:48:05,434 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:48:05,435 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E1E9A926FB8D4B7782C881430107EC86: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:48:05,435 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E1E9A926FB8D4B7782C881430107EC86: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:48:05,435 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs+rhs-cross.txt"]
2016-05-18 01:48:05,435 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B8B4DC5415724A5DB95209889E072D11, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:05,439 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,439 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8B4DC5415724A5DB95209889E072D11, mem on close (mb), free: 126, total: 634, max: 672
2016-05-18 01:48:05,439 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E1E9A926FB8D4B7782C881430107EC86: Starting flush of map output
2016-05-18 01:48:05,439 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E1E9A926FB8D4B7782C881430107EC86: Span0.length = 37, perItem = 24
2016-05-18 01:48:05,441 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E1E9A926FB8D4B7782C881430107EC86: done sorting span=0, length=37, time=1
2016-05-18 01:48:05,441 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E1E9A926FB8D4B7782C881430107EC86: Heap = SpanIterator<0:36> (span=Span[16777216,888]),
2016-05-18 01:48:05,442 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E1E9A926FB8D4B7782C881430107EC86: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-local-dir/output/attempt_1463536085190_0001_1_00_000000_0_10161_0/file.out
2016-05-18 01:48:05,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:05 Completed running task attempt: attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,445 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:48:05,445 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536085190_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:48:05,446 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536085190_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536085435, Process_Duration=4, Process_End_Time=1463536085439, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=37]]
2016-05-18 01:48:05,446 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536085190_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:05,446 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B8B4DC5415724A5DB95209889E072D11, taskAttemptId=attempt_1463536085190_0001_1_00_000000_0, creationTime=1463536085395, allocationTime=1463536085397, startTime=1463536085399, finishTime=1463536085446, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:05,446 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:05,447 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:05,447 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,447 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536085190_0001_00_000001
2016-05-18 01:48:05,447 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536085190_0001_1_00_000000_0
2016-05-18 01:48:05,448 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_FINISHED]: vertexName=B8B4DC5415724A5DB95209889E072D11, taskId=task_1463536085190_0001_1_00_000000, startTime=1463536085399, finishTime=1463536085448, timeTaken=49, status=SUCCEEDED, successfulAttemptID=attempt_1463536085190_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536085435, Process_Duration=4, Process_End_Time=1463536085439, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37
2016-05-18 01:48:05,449 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:05,449 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,449 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_FINISHED]: vertexName=B8B4DC5415724A5DB95209889E072D11, vertexId=vertex_1463536085190_0001_1_00, initRequestedTime=1463536085370, initedTime=1463536085389, startRequestedTime=1463536085377, startedTime=1463536085389, finishTime=1463536085449, timeTaken=60, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=37, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=37, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536085435, Process_Duration=4, Process_End_Time=1463536085439, Read_Duration=2, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, vertexStats=firstTaskStartTime=1463536085399, firstTasksToStart=[ task_1463536085190_0001_1_00_000000 ], lastTaskFinishTime=1463536085446, lastTasksToFinish=[ task_1463536085190_0001_1_00_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536085190_0001_1_00_000000 ], longestDurationTasks=[ task_1463536085190_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:05,449 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:05,450 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536085190_0001_00_000001, stoppedTime=1463536085450, exitStatus=0
2016-05-18 01:48:05,450 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] attempt: attempt_1463536085190_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:48:05,450 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:05,450 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:05,450 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E1E9A926FB8D4B7782C881430107EC86
2016-05-18 01:48:05,451 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536085190_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:48:05,451 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B8B4DC5415724A5DB95209889E072D11, destinationVertexName=E1E9A926FB8D4B7782C881430107EC86. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:48:05,451 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,451 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_STARTED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, taskId=task_1463536085190_0001_1_01_000000, scheduledTime=1463536085451, launchTime=1463536085451
2016-05-18 01:48:05,452 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,453 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536085190_0001_00_000002, launchTime=1463536085453
2016-05-18 01:48:05,454 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536085190_0001_00_000002
2016-05-18 01:48:05,454 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536085190_0001_1_01_000000_0] started. Is using containerId: [container_1463536085190_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:48:05,455 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0, startTime=1463536085454, containerId=container_1463536085190_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:48:05,455 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536085190_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05 Starting to run new task attempt: attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,456 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,456 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,456 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:48:05,457 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true, VertexName: E1E9A926FB8D4B7782C881430107EC86, VertexParallelism: 1, TaskAttemptID:attempt_1463536085190_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B8B4DC5415724A5DB95209889E072D11, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=9C2B51A1E69045068C2451F593151C37, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:48:05,457 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:48:05,457 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,458 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536085190_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:48:05,458 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536085190_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:05,458 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536085190_0001_00_000001, Exiting
2016-05-18 01:48:05,458 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536085190_0001_00_000001
2016-05-18 01:48:05,458 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536085190_0001_00_000001
2016-05-18 01:48:05,460 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536085190_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:05,460 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536085190_0001_00_000001 completed successfully
2016-05-18 01:48:05,460 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536085190_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,461 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536085190_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:48:05,463 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:48:05,466 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:48:05,467 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E1E9A926FB8D4B7782C881430107EC86, ordinal: 1
2016-05-18 01:48:05,468 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1E9A926FB8D4B7782C881430107EC86, mem on start (mb), free: 120, total: 634, max: 672
2016-05-18 01:48:05,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:48:05,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:48:05,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:48:05,473 INFO output.MROutput (MROutput.java:initialize(421)) - 9C2B51A1E69045068C2451F593151C37: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:48:05,473 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:48:05,473 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[9C2B51A1E69045068C2451F593151C37:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B8B4DC5415724A5DB95209889E072D11:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:48:05,473 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:48:05,473 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:48:05,474 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:48:05,474 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[9C2B51A1E69045068C2451F593151C37:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B8B4DC5415724A5DB95209889E072D11:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:48:05,474 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:48:05,474 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B8B4DC5415724A5DB95209889E072D11 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:48:05,474 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B8B4DC5415724A5DB95209889E072D11: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:48:05,474 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:48:05,475 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:48:05,474 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B8B4DC5415724A5DB95209889E072D11: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:48:05,477 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B8B4DC5415724A5DB95209889E072D11 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:48:05,477 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B8B4DC5415724A5DB95209889E072D11
2016-05-18 01:48:05,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:48:05,478 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,479 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8B4DC5415724A5DB95209889E072D11: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:48:05,481 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536085190_0001_1_00_000000_0_10161} to DISK_DIRECT, csize=732, dsize=728, EndTime=1463536085481, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 01:48:05,481 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B8B4DC5415724A5DB95209889E072D11
2016-05-18 01:48:05,481 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:05,481 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B8B4DC5415724A5DB95209889E072D11, wasInterrupted=false
2016-05-18 01:48:05,483 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:48:05,483 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:48:05,483 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(comprehensivesort)[by:[{1}:'num']]
2016-05-18 01:48:05,484 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(comprehensivesort)[by:[{1}:'num']] streamed: true, id: A8B216B562144D48BCE5378399D2A7EF
2016-05-18 01:48:05,484 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true"], id: 9C2B51A1E69045068C2451F593151C37
2016-05-18 01:48:05,484 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001_wd/localmode-local-dir/output/attempt_1463536085190_0001_1_00_000000_0_10161/file.out, len=732, isLocal=true
2016-05-18 01:48:05,484 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 732 bytes from disk
2016-05-18 01:48:05,484 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->['num', 'lower', 'upper']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true"]
2016-05-18 01:48:05,484 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:48:05,484 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(comprehensivesort)[by:[{1}:'num']] A8B216B562144D48BCE5378399D2A7EF, for 1 inputs
2016-05-18 01:48:05,484 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:48:05,485 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - B8B4DC5415724A5DB95209889E072D11: Already shutdown. Ignoring fetch complete
2016-05-18 01:48:05,485 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 712 bytes
2016-05-18 01:48:05,485 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B8B4DC5415724A5DB95209889E072D11
2016-05-18 01:48:05,485 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B8B4DC5415724A5DB95209889E072D11: Shuffle Runner thread complete
2016-05-18 01:48:05,485 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E1E9A926FB8D4B7782C881430107EC86, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:48:05,486 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B8B4DC5415724A5DB95209889E072D11: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:48:05,490 INFO output.MROutput (MROutput.java:close(526)) - 9C2B51A1E69045068C2451F593151C37 closed
2016-05-18 01:48:05,490 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536085190_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:48:05,490 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,490 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1E9A926FB8D4B7782C881430107EC86, mem on close (mb), free: 115, total: 634, max: 672
2016-05-18 01:48:05,491 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B8B4DC5415724A5DB95209889E072D11
2016-05-18 01:48:05,491 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8B4DC5415724A5DB95209889E072D11: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:48:05,491 INFO output.MROutput (MROutput.java:close(526)) - 9C2B51A1E69045068C2451F593151C37 closed
2016-05-18 01:48:05,491 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:48:05 Completed running task attempt: attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,491 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:48:05,491 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536085190_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:48:05,492 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536085190_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536085485, Process_Duration=5, Process_End_Time=1463536085490, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=37]]
2016-05-18 01:48:05,492 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536085190_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:05,492 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:48:05,492 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, taskAttemptId=attempt_1463536085190_0001_1_01_000000_0, creationTime=1463536085452, allocationTime=1463536085452, startTime=1463536085454, finishTime=1463536085492, timeTaken=38, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:48:05,492 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:48:05,492 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,492 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536085190_0001_1_01_000000_0
2016-05-18 01:48:05,493 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:TASK_FINISHED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, taskId=task_1463536085190_0001_1_01_000000, startTime=1463536085454, finishTime=1463536085493, timeTaken=39, status=SUCCEEDED, successfulAttemptID=attempt_1463536085190_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536085485, Process_Duration=5, Process_End_Time=1463536085490, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37
2016-05-18 01:48:05,493 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:48:05,493 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86]
2016-05-18 01:48:05,493 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536085190_0001_00_000002
2016-05-18 01:48:05,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:VERTEX_FINISHED]: vertexName=E1E9A926FB8D4B7782C881430107EC86, vertexId=vertex_1463536085190_0001_1_01, initRequestedTime=1463536085377, initedTime=1463536085381, startRequestedTime=1463536085390, startedTime=1463536085390, finishTime=1463536085493, timeTaken=103, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=37, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=37, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536085485, Process_Duration=5, Process_End_Time=1463536085490, Read_Duration=3, Tuples_Read=37, Tuples_Written=37, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=37, vertexStats=firstTaskStartTime=1463536085454, firstTasksToStart=[ task_1463536085190_0001_1_01_000000 ], lastTaskFinishTime=1463536085492, lastTasksToFinish=[ task_1463536085190_0001_1_01_000000 ], minTaskDuration=38, maxTaskDuration=38, avgTaskDuration=38.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536085190_0001_1_01_000000 ], longestDurationTasks=[ task_1463536085190_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:48:05,494 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:48:05,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536085190_0001_00_000002, stoppedTime=1463536085494, exitStatus=0
2016-05-18 01:48:05,495 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:48:05,495 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:05,495 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536085190_0001_1_00 [B8B4DC5415724A5DB95209889E072D11]
2016-05-18 01:48:05,495 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536085190_0001_1
2016-05-18 01:48:05,495 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536085190_0001_1
2016-05-18 01:48:05,495 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 9C2B51A1E69045068C2451F593151C37 for vertex: vertex_1463536085190_0001_1_01 [E1E9A926FB8D4B7782C881430107EC86], outputName: 9C2B51A1E69045068C2451F593151C37
2016-05-18 01:48:05,497 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536085190_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:48:05,497 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536085190_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:48:05,504 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536085190_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:48:05,504 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536085190_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:48:05,504 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536085190_0001_00_000002, Exiting
2016-05-18 01:48:05,504 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536085190_0001_00_000002
2016-05-18 01:48:05,504 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536085190_0001_00_000002
2016-05-18 01:48:05,504 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536085190_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:48:05,505 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536085190_0001_00_000002 completed successfully
2016-05-18 01:48:05,505 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:9C2B51A1E69045068C2451F593151C37 of vertex/vertexGroup:E1E9A926FB8D4B7782C881430107EC86 isVertexGroupOutput:false
2016-05-18 01:48:05,505 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:48:05,507 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536085190_0001_1, queueSize=0
2016-05-18 01:48:05,507 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536085190_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536085190_0001_1, startTime=1463536085369, finishTime=1463536085506, timeTaken=137, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=290, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=37, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=74, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=74, OUTPUT_BYTES=888, OUTPUT_BYTES_WITH_OVERHEAD=728, OUTPUT_BYTES_PHYSICAL=732, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=732, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=732, SHUFFLE_BYTES_DECOMPRESSED=728, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=732, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072170920, Process_Duration=9, Process_End_Time=2927072170929, Read_Duration=5, Tuples_Read=74, Tuples_Written=74, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=37, Tuples_Written=37
2016-05-18 01:48:05,507 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536085190_0001_1 finished with state: SUCCEEDED
2016-05-18 01:48:05,507 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536085190_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:48:05 Completed Dag: dag_1463536085190_0001_1
2016-05-18 01:48:05,508 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536085190_0001_1, dagState=SUCCEEDED
2016-05-18 01:48:05,508 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536085190_0001_1_post
2016-05-18 01:48:05,508 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:48:05,508 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:48:05,508 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:48:05,508 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true, with id=dag_1463536085190_0001_1
2016-05-18 01:48:05,508 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB] comprehensivesort/(1/1) ...sive/false_true_true_true, with id=dag_1463536085190_0001_1
2016-05-18 01:48:05,514 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...sive/false_true_true_true, applicationId=application_1463536085190_0001
2016-05-18 01:48:05,515 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:48:05,515 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:48:05,515 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:48:05,516 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:48:05,516 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:48:05,516 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:48:05,516 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:48:05,516 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:48:05,517 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:48:05,517 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:48:05,517 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:48:05,517 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:48:05,517 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:48:05,518 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:48:05,521 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/899F193D98B74C9FAE420682B821B078/A86F5486353549BB9B77A9CB3ECDE9EB/.tez/application_1463536085190_0001
2016-05-18 01:48:05,521 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true/_temporary
2016-05-18 01:48:05,522 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [comprehensivesort] completed in: 00:00.375
2016-05-18 01:48:05,588 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_false
2016-05-18 01:48:05,588 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_false
2016-05-18 01:48:05,588 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_true
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_true
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_false
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_true
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_false_true
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_true_false
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_false
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_false
2016-05-18 01:48:05,589 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_false_false
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_false
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_false_true_true
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_false
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_false_true
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_null_true_true
2016-05-18 01:48:05,590 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_false
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_true_false_true
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_false_true
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_true_true
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_false
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_false_true_false
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/false_true_false_true
2016-05-18 01:48:05,591 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/sortedvaluesplatform/testComprehensiveGroupBy/comprehensive/true_null_true_true