2016-05-18 00:38:45,016 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-18 00:38:45,048 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.BufferPipesPlatformTest, with platform: hadoop2-tez
2016-05-18 00:38:45,082 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-18 00:38:45,605 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-18 00:38:45,933 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-18 00:38:46,193 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:46,195 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:46,298 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: 3E55A6CD9DF644AABE68A27A23B52917
2016-05-18 00:38:46,483 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.092
2016-05-18 00:38:46,512 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.122
2016-05-18 00:38:46,513 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:46,514 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:46,516 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:46,687 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:46,823 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-18 00:38:47,041 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-18 00:38:47,044 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:47,044 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:47,045 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-18 00:38:47,045 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:47,045 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:47,045 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:47,046 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...stIdentityBuffer/identity
2016-05-18 00:38:47,047 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:47,065 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:47,232 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:47,239 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:47,239 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:47,287 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001 doesn't exist and is created
2016-05-18 00:38:47,353 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:47,353 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:47,355 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd
2016-05-18 00:38:47,417 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531927238_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:47,431 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:47,432 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:47,442 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:47,442 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:47,442 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:47,444 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:47,454 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:38:47,454 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:38:47,455 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:47,471 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:47,498 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:47,500 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:47,502 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:47,503 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:47,503 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:47,504 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:47,505 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:47,506 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:47,507 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:47,508 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:47,519 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:47,524 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:47,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:47,555 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:38:47,555 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:38:47,555 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:47,555 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:47,556 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:47,560 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-log-dir
2016-05-18 00:38:47,560 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-log-dir/history.txt.appattempt_1463531927238_0001_000000, maxErrors=10
2016-05-18 00:38:47,561 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:47,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531927238_0001_000000, appSubmitTime=1463531927395, launchTime=1463531927415
2016-05-18 00:38:47,568 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:47,630 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:38140
2016-05-18 00:38:47,631 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531927238
2016-05-18 00:38:47,648 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531927238_0001_000000, startTime=1463531927647
2016-05-18 00:38:47,648 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:47,656 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:47,671 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:47,671 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463531927238_0001, dagName=[BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 00:38:47,875 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 00:38:47,884 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:47,941 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531927238_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-log-dir/dag_1463531927238_0001_1.dot
2016-05-18 00:38:47,946 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-log-dir/dag_1463531927238_0001_1-tez-dag.pb.txt
2016-05-18 00:38:47,986 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531927238_0001_1
2016-05-18 00:38:47,986 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:47,986 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 00:38:47 Running Dag: dag_1463531927238_0001_1
2016-05-18 00:38:48,009 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531927238_0001_1, submitTime=1463531927875
2016-05-18 00:38:48,082 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001/tez-conf.pb" } size: 36903 timestamp: 1463531927000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:48,083 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,083 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:48,084 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001/tez-conf.pb" } size: 36903 timestamp: 1463531927000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:48,084 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,085 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:48,093 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:48,097 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531927238_0001_1, initTime=1463531928095
2016-05-18 00:38:48,098 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531927238_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:48,099 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:48,101 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:DAG_STARTED]: dagID=dag_1463531927238_0001_1, startTime=1463531928101
2016-05-18 00:38:48,101 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531927238_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:48,102 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C7983AD5AA2F4B48A1F9589B03B0EEE4 : {7F08EF9F8C104ABFBA8776B564C30733={InputName=7F08EF9F8C104ABFBA8776B564C30733}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:48,102 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 7F08EF9F8C104ABFBA8776B564C30733, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:48,103 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,109 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463531927238_0001, dagName=[BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 00:38:48,110 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,110 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,113 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,120 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 7F08EF9F8C104ABFBA8776B564C30733 on vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,134 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:48,134 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,154 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 7F08EF9F8C104ABFBA8776B564C30733 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:48,163 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531927238_0001, with dag id: dag_1463531927238_0001_1
2016-05-18 00:38:48,168 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:48,177 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,192 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:48,193 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,193 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:48,194 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:48,196 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:48,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531927238_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:48,197 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD], numAdditionalOutputs=1
2016-05-18 00:38:48,198 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=62558A51EA3D47DC84EF3BBB35D0918B, vertex=vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:48,208 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C5EDB084B8084E6FA08D9BD0E28535CD:62558A51EA3D47DC84EF3BBB35D0918B using oldmapred API
2016-05-18 00:38:48,209 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=62558A51EA3D47DC84EF3BBB35D0918B, vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:48,210 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 7F08EF9F8C104ABFBA8776B564C30733 on vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, vertexId=vertex_1463531927238_0001_1_01, initRequestedTime=1463531928134, initedTime=1463531928197, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:48,221 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:48,226 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:48,228 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {7F08EF9F8C104ABFBA8776B564C30733=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:48,229 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] parallelism set to 1
2016-05-18 00:38:48,229 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:48,240 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, vertexId=vertex_1463531927238_0001_1_00, initRequestedTime=1463531928102, initedTime=1463531928240, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:48,240 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] with distanceFromRoot: 0
2016-05-18 00:38:48,240 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:48,241 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531927238_0001_1_00, startRequestedTime=1463531928134, startedTime=1463531928240
2016-05-18 00:38:48,242 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531927238_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:48,242 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:48,243 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531927238_0001_1_00 for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] numStartedSources: 1 numSources: 1
2016-05-18 00:38:48,243 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] with distanceFromRoot: 1
2016-05-18 00:38:48,244 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C7983AD5AA2F4B48A1F9589B03B0EEE4
2016-05-18 00:38:48,244 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531927238_0001_1_01, startRequestedTime=1463531928243, startedTime=1463531928244
2016-05-18 00:38:48,245 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:48,249 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,256 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_STARTED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, taskId=task_1463531927238_0001_1_00_000000, scheduledTime=1463531928255, launchTime=1463531928255
2016-05-18 00:38:48,259 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C5EDB084B8084E6FA08D9BD0E28535CD with 0 source tasks and 1 pending tasks
2016-05-18 00:38:48,266 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C7983AD5AA2F4B48A1F9589B03B0EEE4 in vertex: C5EDB084B8084E6FA08D9BD0E28535CD numBipartiteSourceTasks: 1
2016-05-18 00:38:48,280 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,320 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:48,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531927238_0001_00_000001, launchTime=1463531928336
2016-05-18 00:38:48,343 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531927238_0001_00_000001
2016-05-18 00:38:48,348 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531927238_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,348 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531927238_0001_1_00_000000_0] started. Is using containerId: [container_1463531927238_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:48,349 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0, startTime=1463531928346, containerId=container_1463531927238_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:48 Starting to run new task attempt: attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,350 INFO counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-18 00:38:48,351 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,352 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:48,354 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:48,365 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity, VertexName: C7983AD5AA2F4B48A1F9589B03B0EEE4, VertexParallelism: 1, TaskAttemptID:attempt_1463531927238_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=7F08EF9F8C104ABFBA8776B564C30733, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C5EDB084B8084E6FA08D9BD0E28535CD, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:48,367 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:48,372 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,379 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531927238_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:48,382 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531927238_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:48,408 INFO input.MRInput (MRInput.java:initialize(440)) - 7F08EF9F8C104ABFBA8776B564C30733 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:48,409 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 7F08EF9F8C104ABFBA8776B564C30733
2016-05-18 00:38:48,410 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:48,422 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C7983AD5AA2F4B48A1F9589B03B0EEE4, ordinal: 0
2016-05-18 00:38:48,423 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7983AD5AA2F4B48A1F9589B03B0EEE4, mem on start (mb), free: 224, total: 281, max: 672
2016-05-18 00:38:48,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:48,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:48,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:48,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:48,425 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[7F08EF9F8C104ABFBA8776B564C30733:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C5EDB084B8084E6FA08D9BD0E28535CD:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:48,430 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:48,432 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:48,433 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:48,433 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[7F08EF9F8C104ABFBA8776B564C30733:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C5EDB084B8084E6FA08D9BD0E28535CD:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:48,434 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:48,435 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 7F08EF9F8C104ABFBA8776B564C30733
2016-05-18 00:38:48,435 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 7F08EF9F8C104ABFBA8776B564C30733 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:48,436 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:48,436 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:48,437 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:48,439 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,460 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:48,461 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 7F08EF9F8C104ABFBA8776B564C30733 initialized RecordReader from event
2016-05-18 00:38:48,485 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:48,519 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 7F08EF9F8C104ABFBA8776B564C30733
2016-05-18 00:38:48,519 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: F370677F8EB6419CA526782451983CF3
2016-05-18 00:38:48,521 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] F370677F8EB6419CA526782451983CF3
2016-05-18 00:38:48,564 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C5EDB084B8084E6FA08D9BD0E28535CD using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6ed25fb4, comparator=cascading.tuple.hadoop.util.TupleComparator@35895b43, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:48,571 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C5EDB084B8084E6FA08D9BD0E28535CD: , UsingHashComparator=false
2016-05-18 00:38:48,686 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:48,686 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:48,686 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C5EDB084B8084E6FA08D9BD0E28535CD: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:48,689 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C5EDB084B8084E6FA08D9BD0E28535CD: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:48,694 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:48,696 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C7983AD5AA2F4B48A1F9589B03B0EEE4, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:38:48,705 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,705 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7983AD5AA2F4B48A1F9589B03B0EEE4, mem on close (mb), free: 112, total: 281, max: 672
2016-05-18 00:38:48,706 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C5EDB084B8084E6FA08D9BD0E28535CD: Starting flush of map output
2016-05-18 00:38:48,707 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C5EDB084B8084E6FA08D9BD0E28535CD: Span0.length = 13, perItem = 15
2016-05-18 00:38:48,709 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C5EDB084B8084E6FA08D9BD0E28535CD: done sorting span=0, length=13, time=2
2016-05-18 00:38:48,712 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C5EDB084B8084E6FA08D9BD0E28535CD: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 00:38:48,718 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C5EDB084B8084E6FA08D9BD0E28535CD: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-local-dir/output/attempt_1463531927238_0001_1_00_000000_0_10003_0/file.out
2016-05-18 00:38:48,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:48 Completed running task attempt: attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,748 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:48,749 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531927238_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:48,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531927238_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531928696, Process_Duration=9, Process_End_Time=1463531928705, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:48,756 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531927238_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:48,757 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, taskAttemptId=attempt_1463531927238_0001_1_00_000000_0, creationTime=1463531928275, allocationTime=1463531928330, startTime=1463531928346, finishTime=1463531928755, timeTaken=409, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:48,758 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:48,762 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:48,762 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:48,763 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531927238_0001_00_000001
2016-05-18 00:38:48,767 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531927238_0001_1_00_000000_0
2016-05-18 00:38:48,773 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_FINISHED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, taskId=task_1463531927238_0001_1_00_000000, startTime=1463531928346, finishTime=1463531928768, timeTaken=422, status=SUCCEEDED, successfulAttemptID=attempt_1463531927238_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531928696, Process_Duration=9, Process_End_Time=1463531928705, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:48,780 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:48,780 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,796 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_FINISHED]: vertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, vertexId=vertex_1463531927238_0001_1_00, initRequestedTime=1463531928102, initedTime=1463531928240, startRequestedTime=1463531928134, startedTime=1463531928240, finishTime=1463531928780, timeTaken=540, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531928696, Process_Duration=9, Process_End_Time=1463531928705, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531928346, firstTasksToStart=[ task_1463531927238_0001_1_00_000000 ], lastTaskFinishTime=1463531928755, lastTasksToFinish=[ task_1463531927238_0001_1_00_000000 ], minTaskDuration=409, maxTaskDuration=409, avgTaskDuration=409.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531927238_0001_1_00_000000 ], longestDurationTasks=[ task_1463531927238_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:48,797 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:48,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531927238_0001_00_000001, stoppedTime=1463531928803, exitStatus=0
2016-05-18 00:38:48,804 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] attempt: attempt_1463531927238_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:48,805 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C5EDB084B8084E6FA08D9BD0E28535CD
2016-05-18 00:38:48,806 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, destinationVertexName=C5EDB084B8084E6FA08D9BD0E28535CD. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:48,806 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,807 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:48,807 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:48,808 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531927238_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:48,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_STARTED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, taskId=task_1463531927238_0001_1_01_000000, scheduledTime=1463531928808, launchTime=1463531928808
2016-05-18 00:38:48,809 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,811 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531927238_0001_00_000002, launchTime=1463531928810
2016-05-18 00:38:48,812 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531927238_0001_00_000002
2016-05-18 00:38:48,813 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531927238_0001_1_01_000000_0] started. Is using containerId: [container_1463531927238_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:48,813 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0, startTime=1463531928812, containerId=container_1463531927238_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:48,815 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531927238_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48 Starting to run new task attempt: attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,816 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,816 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:48,816 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:48,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity, VertexName: C5EDB084B8084E6FA08D9BD0E28535CD, VertexParallelism: 1, TaskAttemptID:attempt_1463531927238_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C7983AD5AA2F4B48A1F9589B03B0EEE4, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=62558A51EA3D47DC84EF3BBB35D0918B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:48,819 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:48,820 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,823 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531927238_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,824 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531927238_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:48,829 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531927238_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:48,829 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531927238_0001_00_000001: 66 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:48,829 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531927238_0001_00_000001, Exiting
2016-05-18 00:38:48,829 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531927238_0001_00_000001
2016-05-18 00:38:48,830 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531927238_0001_00_000001
2016-05-18 00:38:48,833 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:48,838 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531927238_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:48,838 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531927238_0001_00_000001 completed successfully
2016-05-18 00:38:48,856 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:48,863 INFO output.MROutput (MROutput.java:initialize(421)) - 62558A51EA3D47DC84EF3BBB35D0918B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:48,863 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C5EDB084B8084E6FA08D9BD0E28535CD, ordinal: 1
2016-05-18 00:38:48,864 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C5EDB084B8084E6FA08D9BD0E28535CD, mem on start (mb), free: 100, total: 281, max: 672
2016-05-18 00:38:48,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:48,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:48,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:48,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:48,867 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[62558A51EA3D47DC84EF3BBB35D0918B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C7983AD5AA2F4B48A1F9589B03B0EEE4:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:38:48,867 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:48,868 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:48,868 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:48,868 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[62558A51EA3D47DC84EF3BBB35D0918B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C7983AD5AA2F4B48A1F9589B03B0EEE4:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:38:48,869 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:48,870 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C7983AD5AA2F4B48A1F9589B03B0EEE4 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:48,870 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:48,871 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:48,871 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:48,873 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:48,888 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C7983AD5AA2F4B48A1F9589B03B0EEE4 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:48,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C7983AD5AA2F4B48A1F9589B03B0EEE4
2016-05-18 00:38:48,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:48,896 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,901 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:48,902 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:48,909 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: F370677F8EB6419CA526782451983CF3
2016-05-18 00:38:48,910 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"], id: 62558A51EA3D47DC84EF3BBB35D0918B
2016-05-18 00:38:48,910 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-18 00:38:48,911 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] F370677F8EB6419CA526782451983CF3, for 1 inputs
2016-05-18 00:38:48,922 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531927238_0001_1_00_000000_0_10003} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463531928922, TimeTaken=5, Rate=0.03 MB/s
2016-05-18 00:38:48,923 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C7983AD5AA2F4B48A1F9589B03B0EEE4
2016-05-18 00:38:48,923 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:48,924 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C7983AD5AA2F4B48A1F9589B03B0EEE4, wasInterrupted=false
2016-05-18 00:38:48,925 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:48,926 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: Already shutdown. Ignoring fetch complete
2016-05-18 00:38:48,928 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:48,929 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001_wd/localmode-local-dir/output/attempt_1463531927238_0001_1_00_000000_0_10003/file.out, len=175, isLocal=true
2016-05-18 00:38:48,931 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 00:38:48,931 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:48,936 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:48,938 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 00:38:48,939 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C7983AD5AA2F4B48A1F9589B03B0EEE4
2016-05-18 00:38:48,939 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C5EDB084B8084E6FA08D9BD0E28535CD, all 1 inputs ready in: 00:00:00.028
2016-05-18 00:38:48,939 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: Shuffle Runner thread complete
2016-05-18 00:38:48,941 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:48,953 INFO output.MROutput (MROutput.java:close(526)) - 62558A51EA3D47DC84EF3BBB35D0918B closed
2016-05-18 00:38:48,954 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531927238_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:48,955 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,955 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C5EDB084B8084E6FA08D9BD0E28535CD, mem on close (mb), free: 93, total: 281, max: 672
2016-05-18 00:38:48,955 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C7983AD5AA2F4B48A1F9589B03B0EEE4
2016-05-18 00:38:48,956 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C7983AD5AA2F4B48A1F9589B03B0EEE4: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:48,956 INFO output.MROutput (MROutput.java:close(526)) - 62558A51EA3D47DC84EF3BBB35D0918B closed
2016-05-18 00:38:48,956 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:48 Completed running task attempt: attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,957 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:48,958 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531927238_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:48,959 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531927238_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=53, MERGE_PHASE_TIME=68, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=31][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531928939, Process_Duration=16, Process_End_Time=1463531928955, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=13]]
2016-05-18 00:38:48,961 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531927238_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:48,961 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, taskAttemptId=attempt_1463531927238_0001_1_01_000000_0, creationTime=1463531928808, allocationTime=1463531928810, startTime=1463531928812, finishTime=1463531928961, timeTaken=149, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:48,961 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:48,963 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:48,963 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:48,963 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531927238_0001_00_000002
2016-05-18 00:38:48,962 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531927238_0001_1_01_000000_0
2016-05-18 00:38:48,966 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:TASK_FINISHED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, taskId=task_1463531927238_0001_1_01_000000, startTime=1463531928812, finishTime=1463531928964, timeTaken=152, status=SUCCEEDED, successfulAttemptID=attempt_1463531927238_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=53, MERGE_PHASE_TIME=68, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=31, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531928939, Process_Duration=16, Process_End_Time=1463531928955, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=13
2016-05-18 00:38:48,966 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:48,966 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD]
2016-05-18 00:38:48,969 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:VERTEX_FINISHED]: vertexName=C5EDB084B8084E6FA08D9BD0E28535CD, vertexId=vertex_1463531927238_0001_1_01, initRequestedTime=1463531928134, initedTime=1463531928197, startRequestedTime=1463531928243, startedTime=1463531928244, finishTime=1463531928967, timeTaken=723, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=53, MERGE_PHASE_TIME=68, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=31, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531928939, Process_Duration=16, Process_End_Time=1463531928955, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=13, vertexStats=firstTaskStartTime=1463531928812, firstTasksToStart=[ task_1463531927238_0001_1_01_000000 ], lastTaskFinishTime=1463531928961, lastTasksToFinish=[ task_1463531927238_0001_1_01_000000 ], minTaskDuration=149, maxTaskDuration=149, avgTaskDuration=149.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531927238_0001_1_01_000000 ], longestDurationTasks=[ task_1463531927238_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:48,969 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:48,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531927238_0001_00_000002, stoppedTime=1463531928969, exitStatus=0
2016-05-18 00:38:48,970 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:48,970 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:48,970 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531927238_0001_1_00 [C7983AD5AA2F4B48A1F9589B03B0EEE4]
2016-05-18 00:38:48,972 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531927238_0001_1
2016-05-18 00:38:48,973 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531927238_0001_1
2016-05-18 00:38:48,974 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 62558A51EA3D47DC84EF3BBB35D0918B for vertex: vertex_1463531927238_0001_1_01 [C5EDB084B8084E6FA08D9BD0E28535CD], outputName: 62558A51EA3D47DC84EF3BBB35D0918B
2016-05-18 00:38:48,980 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531927238_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:48,981 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531927238_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:48,981 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531927238_0001_00_000002: 17 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:48,982 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531927238_0001_00_000002, Exiting
2016-05-18 00:38:48,982 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531927238_0001_00_000002
2016-05-18 00:38:48,982 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531927238_0001_00_000002
2016-05-18 00:38:48,982 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531927238_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:48,983 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531927238_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:48,983 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531927238_0001_00_000002 completed successfully
2016-05-18 00:38:48,988 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:62558A51EA3D47DC84EF3BBB35D0918B of vertex/vertexGroup:C5EDB084B8084E6FA08D9BD0E28535CD isVertexGroupOutput:false
2016-05-18 00:38:48,988 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:48,996 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531927238_0001_1, queueSize=0
2016-05-18 00:38:48,999 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531927238_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531927238_0001_1, startTime=1463531928099, finishTime=1463531928989, timeTaken=890, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=2090, AM_GC_TIME_MILLIS=24, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=26, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=53, MERGE_PHASE_TIME=68, FIRST_EVENT_RECEIVED=31, LAST_EVENT_RECEIVED=31, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063857635, Process_Duration=25, Process_End_Time=2927063857660, Read_Duration=3, Tuples_Read=26, Tuples_Written=26, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=13
2016-05-18 00:38:49,000 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531927238_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:49,000 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531927238_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:49 Completed Dag: dag_1463531927238_0001_1
2016-05-18 00:38:49,002 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531927238_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:49,003 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531927238_0001_1_post
2016-05-18 00:38:49,003 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:49,005 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:49,005 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:49,007 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity, with id=dag_1463531927238_0001_1
2016-05-18 00:38:49,011 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E] test/(1/1) ...stIdentityBuffer/identity, with id=dag_1463531927238_0001_1
2016-05-18 00:38:49,013 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463531927238_0001
2016-05-18 00:38:49,019 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:49,019 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:49,019 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:49,019 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:49,019 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:49,022 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:49,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:49,023 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:49,023 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:49,023 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:49,024 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:49,024 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:49,024 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:49,025 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:49,029 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF7E387DB9064C0CBDAA2C4D6FB560E9/EE1B645C4557499F91BB672255AA618E/.tez/application_1463531927238_0001
2016-05-18 00:38:49,032 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity/_temporary
2016-05-18 00:38:49,033 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:01.987
2016-05-18 00:38:49,162 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-18 00:38:49,241 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:49,241 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:49,386 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.054
2016-05-18 00:38:49,403 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.070
2016-05-18 00:38:49,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:49,405 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:49,405 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:49,427 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:49,448 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-18 00:38:49,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:49,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:49,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-18 00:38:49,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:49,527 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:49,528 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:49,530 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferForward/firstn
2016-05-18 00:38:49,530 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:49,553 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:49,558 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:49,558 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:49,559 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:49,569 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001 doesn't exist and is created
2016-05-18 00:38:49,590 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:49,590 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:49,591 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd
2016-05-18 00:38:49,603 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531929558_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:49,604 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:49,604 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:49,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:49,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:49,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:49,607 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:49,608 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:49,611 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:49,612 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:49,612 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:49,612 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:49,612 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:49,612 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:49,613 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:49,613 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:49,613 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:49,613 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:49,614 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:49,614 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:49,617 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:49,618 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:49,618 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:49,618 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-log-dir
2016-05-18 00:38:49,619 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-log-dir/history.txt.appattempt_1463531929558_0001_000000, maxErrors=10
2016-05-18 00:38:49,619 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:49,619 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531929558_0001_000000, appSubmitTime=1463531929602, launchTime=1463531929603
2016-05-18 00:38:49,632 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:55823
2016-05-18 00:38:49,635 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:49,636 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531929558
2016-05-18 00:38:49,644 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531929558_0001_000000, startTime=1463531929644
2016-05-18 00:38:49,644 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:49,690 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:49,690 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:49,691 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463531929558_0001, dagName=[68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 00:38:49,716 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 00:38:49,717 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:49,718 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531929558_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-log-dir/dag_1463531929558_0001_1.dot
2016-05-18 00:38:49,720 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-log-dir/dag_1463531929558_0001_1-tez-dag.pb.txt
2016-05-18 00:38:49,727 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531929558_0001_1
2016-05-18 00:38:49,728 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:49,728 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 00:38:49 Running Dag: dag_1463531929558_0001_1
2016-05-18 00:38:49,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531929558_0001_1, submitTime=1463531929716
2016-05-18 00:38:49,740 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001/tez-conf.pb" } size: 36903 timestamp: 1463531929000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:49,740 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,741 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:49,742 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001/tez-conf.pb" } size: 36903 timestamp: 1463531929000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:49,742 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,742 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:49,743 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:49,744 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531929558_0001_1, initTime=1463531929743
2016-05-18 00:38:49,744 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531929558_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:49,746 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:49,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:DAG_STARTED]: dagID=dag_1463531929558_0001_1, startTime=1463531929746
2016-05-18 00:38:49,747 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531929558_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:49,747 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C47C7AED650041BE8B3E16EFED76EE5E : {67626284826A486E87B3C1383959404B={InputName=67626284826A486E87B3C1383959404B}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:49,747 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 67626284826A486E87B3C1383959404B, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:49,747 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,748 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,748 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,748 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,748 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463531929558_0001, dagName=[68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 00:38:49,749 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:49,749 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,750 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 67626284826A486E87B3C1383959404B on vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,757 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 67626284826A486E87B3C1383959404B asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:49,761 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:49,762 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:49,762 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:49,763 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:49,767 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 67626284826A486E87B3C1383959404B on vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,770 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:49,770 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,771 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,772 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531929558_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:49,773 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818], numAdditionalOutputs=1
2016-05-18 00:38:49,773 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=86A9A9EC012D4868BB1F5EDFD40A2A9E, vertex=vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:49,776 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A680905727274BA4B95D68B330AC2818:86A9A9EC012D4868BB1F5EDFD40A2A9E using oldmapred API
2016-05-18 00:38:49,777 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=86A9A9EC012D4868BB1F5EDFD40A2A9E, vertexName=A680905727274BA4B95D68B330AC2818, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:49,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A680905727274BA4B95D68B330AC2818, vertexId=vertex_1463531929558_0001_1_01, initRequestedTime=1463531929749, initedTime=1463531929773, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:49,778 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:49,781 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:49,781 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {67626284826A486E87B3C1383959404B=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:49,782 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] parallelism set to 1
2016-05-18 00:38:49,782 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:49,783 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, vertexId=vertex_1463531929558_0001_1_00, initRequestedTime=1463531929747, initedTime=1463531929783, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:49,784 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] with distanceFromRoot: 0
2016-05-18 00:38:49,784 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:49,785 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531929558_0001, with dag id: dag_1463531929558_0001_1
2016-05-18 00:38:49,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531929558_0001_1_00, startRequestedTime=1463531929749, startedTime=1463531929784
2016-05-18 00:38:49,786 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531929558_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:49,787 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:49,787 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531929558_0001_1_00 for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] numStartedSources: 1 numSources: 1
2016-05-18 00:38:49,787 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] with distanceFromRoot: 1
2016-05-18 00:38:49,787 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C47C7AED650041BE8B3E16EFED76EE5E
2016-05-18 00:38:49,788 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531929558_0001_1_01, startRequestedTime=1463531929787, startedTime=1463531929788
2016-05-18 00:38:49,789 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:49,791 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A680905727274BA4B95D68B330AC2818 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:49,792 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C47C7AED650041BE8B3E16EFED76EE5E in vertex: A680905727274BA4B95D68B330AC2818 numBipartiteSourceTasks: 1
2016-05-18 00:38:49,793 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_STARTED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, taskId=task_1463531929558_0001_1_00_000000, scheduledTime=1463531929792, launchTime=1463531929792
2016-05-18 00:38:49,794 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,796 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:49,798 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531929558_0001_00_000001, launchTime=1463531929798
2016-05-18 00:38:49,799 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531929558_0001_00_000001
2016-05-18 00:38:49,800 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531929558_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,800 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531929558_0001_1_00_000000_0] started. Is using containerId: [container_1463531929558_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:49,804 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0, startTime=1463531929800, containerId=container_1463531929558_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:49 Starting to run new task attempt: attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,805 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,805 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:49,805 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:49,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn, VertexName: C47C7AED650041BE8B3E16EFED76EE5E, VertexParallelism: 1, TaskAttemptID:attempt_1463531929558_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=67626284826A486E87B3C1383959404B, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A680905727274BA4B95D68B330AC2818, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:49,806 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:49,807 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,808 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531929558_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:49,809 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531929558_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:49,815 INFO input.MRInput (MRInput.java:initialize(440)) - 67626284826A486E87B3C1383959404B using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:49,815 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 67626284826A486E87B3C1383959404B
2016-05-18 00:38:49,816 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:49,824 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C47C7AED650041BE8B3E16EFED76EE5E, ordinal: 0
2016-05-18 00:38:49,824 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C47C7AED650041BE8B3E16EFED76EE5E, mem on start (mb), free: 89, total: 281, max: 672
2016-05-18 00:38:49,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:49,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:49,826 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:49,826 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:49,826 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[67626284826A486E87B3C1383959404B:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A680905727274BA4B95D68B330AC2818:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:49,827 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:49,827 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:49,828 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:49,828 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[67626284826A486E87B3C1383959404B:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A680905727274BA4B95D68B330AC2818:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:49,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:49,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 67626284826A486E87B3C1383959404B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:49,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 67626284826A486E87B3C1383959404B
2016-05-18 00:38:49,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:49,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:49,831 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:49,831 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,833 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:49,833 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 67626284826A486E87B3C1383959404B initialized RecordReader from event
2016-05-18 00:38:49,836 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:49,838 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 67626284826A486E87B3C1383959404B
2016-05-18 00:38:49,839 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 1189404BCB5D46A5B974477D30CDB904
2016-05-18 00:38:49,839 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 1189404BCB5D46A5B974477D30CDB904
2016-05-18 00:38:49,844 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A680905727274BA4B95D68B330AC2818 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2166f93f, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@6f75a83a, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:49,846 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A680905727274BA4B95D68B330AC2818: , UsingHashComparator=false
2016-05-18 00:38:49,957 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:49,958 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:49,958 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A680905727274BA4B95D68B330AC2818: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:49,958 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A680905727274BA4B95D68B330AC2818: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:49,958 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:49,959 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C47C7AED650041BE8B3E16EFED76EE5E, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:49,963 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,963 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C47C7AED650041BE8B3E16EFED76EE5E, mem on close (mb), free: 87, total: 382, max: 672
2016-05-18 00:38:49,963 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A680905727274BA4B95D68B330AC2818: Starting flush of map output
2016-05-18 00:38:49,963 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A680905727274BA4B95D68B330AC2818: Span0.length = 13, perItem = 22
2016-05-18 00:38:49,964 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A680905727274BA4B95D68B330AC2818: done sorting span=0, length=13, time=0
2016-05-18 00:38:49,964 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A680905727274BA4B95D68B330AC2818: Heap = SpanIterator<0:12> (span=Span[16777216,286]),
2016-05-18 00:38:49,965 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A680905727274BA4B95D68B330AC2818: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-local-dir/output/attempt_1463531929558_0001_1_00_000000_0_10009_0/file.out
2016-05-18 00:38:49,971 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:49 Completed running task attempt: attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,971 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:49,972 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531929558_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:49,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531929558_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531929959, Process_Duration=4, Process_End_Time=1463531929963, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:49,973 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531929558_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:49,973 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:49,974 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:49,974 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:49,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, taskAttemptId=attempt_1463531929558_0001_1_00_000000_0, creationTime=1463531929794, allocationTime=1463531929797, startTime=1463531929800, finishTime=1463531929973, timeTaken=173, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:49,974 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531929558_0001_00_000001
2016-05-18 00:38:49,975 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531929558_0001_1_00_000000_0
2016-05-18 00:38:49,977 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_FINISHED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, taskId=task_1463531929558_0001_1_00_000000, startTime=1463531929800, finishTime=1463531929975, timeTaken=175, status=SUCCEEDED, successfulAttemptID=attempt_1463531929558_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531929959, Process_Duration=4, Process_End_Time=1463531929963, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:49,977 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:49,977 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:49,979 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_FINISHED]: vertexName=C47C7AED650041BE8B3E16EFED76EE5E, vertexId=vertex_1463531929558_0001_1_00, initRequestedTime=1463531929747, initedTime=1463531929783, startRequestedTime=1463531929749, startedTime=1463531929784, finishTime=1463531929977, timeTaken=193, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531929959, Process_Duration=4, Process_End_Time=1463531929963, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531929800, firstTasksToStart=[ task_1463531929558_0001_1_00_000000 ], lastTaskFinishTime=1463531929973, lastTasksToFinish=[ task_1463531929558_0001_1_00_000000 ], minTaskDuration=173, maxTaskDuration=173, avgTaskDuration=173.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531929558_0001_1_00_000000 ], longestDurationTasks=[ task_1463531929558_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:49,979 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:49,980 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] attempt: attempt_1463531929558_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:49,980 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A680905727274BA4B95D68B330AC2818
2016-05-18 00:38:49,981 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C47C7AED650041BE8B3E16EFED76EE5E, destinationVertexName=A680905727274BA4B95D68B330AC2818. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:49,981 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531929558_0001_00_000001, stoppedTime=1463531929980, exitStatus=0
2016-05-18 00:38:49,981 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:49,982 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:49,981 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,982 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531929558_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:49,983 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_STARTED]: vertexName=A680905727274BA4B95D68B330AC2818, taskId=task_1463531929558_0001_1_01_000000, scheduledTime=1463531929983, launchTime=1463531929983
2016-05-18 00:38:49,983 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:49,987 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531929558_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:49,987 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531929558_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:49,987 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531929558_0001_00_000001, Exiting
2016-05-18 00:38:49,988 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531929558_0001_00_000001
2016-05-18 00:38:49,988 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531929558_0001_00_000001
2016-05-18 00:38:49,990 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531929558_0001_00_000002, launchTime=1463531929988
2016-05-18 00:38:49,991 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531929558_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:49,991 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531929558_0001_00_000002
2016-05-18 00:38:49,991 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531929558_0001_00_000001 completed successfully
2016-05-18 00:38:49,992 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531929558_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:49,992 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531929558_0001_1_01_000000_0] started. Is using containerId: [container_1463531929558_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:49,992 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A680905727274BA4B95D68B330AC2818, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0, startTime=1463531929992, containerId=container_1463531929558_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:49 Starting to run new task attempt: attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:49,993 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:49,993 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:49,993 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:49,994 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn, VertexName: A680905727274BA4B95D68B330AC2818, VertexParallelism: 1, TaskAttemptID:attempt_1463531929558_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C47C7AED650041BE8B3E16EFED76EE5E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=86A9A9EC012D4868BB1F5EDFD40A2A9E, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:49,994 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:49,994 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:49,997 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531929558_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:50,002 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531929558_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:50,010 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:50,010 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:50,020 INFO output.MROutput (MROutput.java:initialize(421)) - 86A9A9EC012D4868BB1F5EDFD40A2A9E: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:50,028 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A680905727274BA4B95D68B330AC2818, ordinal: 1
2016-05-18 00:38:50,028 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A680905727274BA4B95D68B330AC2818, mem on start (mb), free: 80, total: 382, max: 672
2016-05-18 00:38:50,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:50,029 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:50,029 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:50,030 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:50,030 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[86A9A9EC012D4868BB1F5EDFD40A2A9E:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C47C7AED650041BE8B3E16EFED76EE5E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:38:50,031 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:50,031 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:50,032 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:50,032 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[86A9A9EC012D4868BB1F5EDFD40A2A9E:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C47C7AED650041BE8B3E16EFED76EE5E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:38:50,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:50,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C47C7AED650041BE8B3E16EFED76EE5E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:50,034 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C47C7AED650041BE8B3E16EFED76EE5E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:50,034 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C47C7AED650041BE8B3E16EFED76EE5E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:50,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:50,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:50,036 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C47C7AED650041BE8B3E16EFED76EE5E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:50,038 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C47C7AED650041BE8B3E16EFED76EE5E
2016-05-18 00:38:50,038 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:50,040 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:50,042 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:50,043 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 1189404BCB5D46A5B974477D30CDB904
2016-05-18 00:38:50,043 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C47C7AED650041BE8B3E16EFED76EE5E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:50,043 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"], id: 86A9A9EC012D4868BB1F5EDFD40A2A9E
2016-05-18 00:38:50,044 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-18 00:38:50,044 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 1189404BCB5D46A5B974477D30CDB904, for 1 inputs
2016-05-18 00:38:50,046 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531929558_0001_1_00_000000_0_10009} to DISK_DIRECT, csize=322, dsize=318, EndTime=1463531930046, TimeTaken=1, Rate=0.31 MB/s
2016-05-18 00:38:50,046 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C47C7AED650041BE8B3E16EFED76EE5E
2016-05-18 00:38:50,046 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:50,047 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C47C7AED650041BE8B3E16EFED76EE5E, wasInterrupted=false
2016-05-18 00:38:50,047 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:50,048 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:50,049 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001_wd/localmode-local-dir/output/attempt_1463531929558_0001_1_00_000000_0_10009/file.out, len=322, isLocal=true
2016-05-18 00:38:50,049 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 322 bytes from disk
2016-05-18 00:38:50,049 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:50,050 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:50,050 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-18 00:38:50,051 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A680905727274BA4B95D68B330AC2818, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:38:50,051 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C47C7AED650041BE8B3E16EFED76EE5E
2016-05-18 00:38:50,052 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C47C7AED650041BE8B3E16EFED76EE5E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:50,053 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C47C7AED650041BE8B3E16EFED76EE5E: Shuffle Runner thread complete
2016-05-18 00:38:50,058 INFO output.MROutput (MROutput.java:close(526)) - 86A9A9EC012D4868BB1F5EDFD40A2A9E closed
2016-05-18 00:38:50,058 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531929558_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:50,059 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:50,059 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A680905727274BA4B95D68B330AC2818, mem on close (mb), free: 77, total: 382, max: 672
2016-05-18 00:38:50,060 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C47C7AED650041BE8B3E16EFED76EE5E
2016-05-18 00:38:50,060 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C47C7AED650041BE8B3E16EFED76EE5E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:50,060 INFO output.MROutput (MROutput.java:close(526)) - 86A9A9EC012D4868BB1F5EDFD40A2A9E closed
2016-05-18 00:38:50,060 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:50 Completed running task attempt: attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:50,062 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:50,062 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531929558_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:50,063 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531929558_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531930051, Process_Duration=8, Process_End_Time=1463531930059, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 00:38:50,065 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531929558_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,065 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,065 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A680905727274BA4B95D68B330AC2818, taskAttemptId=attempt_1463531929558_0001_1_01_000000_0, creationTime=1463531929983, allocationTime=1463531929987, startTime=1463531929992, finishTime=1463531930065, timeTaken=73, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:50,065 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:50,067 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,067 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531929558_0001_1_01_000000_0
2016-05-18 00:38:50,068 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531929558_0001_00_000002
2016-05-18 00:38:50,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:TASK_FINISHED]: vertexName=A680905727274BA4B95D68B330AC2818, taskId=task_1463531929558_0001_1_01_000000, startTime=1463531929992, finishTime=1463531930068, timeTaken=76, status=SUCCEEDED, successfulAttemptID=attempt_1463531929558_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531930051, Process_Duration=8, Process_End_Time=1463531930059, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 00:38:50,070 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:50,070 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818]
2016-05-18 00:38:50,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:VERTEX_FINISHED]: vertexName=A680905727274BA4B95D68B330AC2818, vertexId=vertex_1463531929558_0001_1_01, initRequestedTime=1463531929749, initedTime=1463531929773, startRequestedTime=1463531929787, startedTime=1463531929788, finishTime=1463531930070, timeTaken=282, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531930051, Process_Duration=8, Process_End_Time=1463531930059, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463531929992, firstTasksToStart=[ task_1463531929558_0001_1_01_000000 ], lastTaskFinishTime=1463531930065, lastTasksToFinish=[ task_1463531929558_0001_1_01_000000 ], minTaskDuration=73, maxTaskDuration=73, avgTaskDuration=73.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531929558_0001_1_01_000000 ], longestDurationTasks=[ task_1463531929558_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:50,072 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:50,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531929558_0001_00_000002, stoppedTime=1463531930073, exitStatus=0
2016-05-18 00:38:50,073 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:50,073 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:50,074 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531929558_0001_1_00 [C47C7AED650041BE8B3E16EFED76EE5E]
2016-05-18 00:38:50,074 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531929558_0001_1
2016-05-18 00:38:50,074 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531929558_0001_1
2016-05-18 00:38:50,075 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 86A9A9EC012D4868BB1F5EDFD40A2A9E for vertex: vertex_1463531929558_0001_1_01 [A680905727274BA4B95D68B330AC2818], outputName: 86A9A9EC012D4868BB1F5EDFD40A2A9E
2016-05-18 00:38:50,075 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531929558_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:50,082 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531929558_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:50,083 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531929558_0001_00_000002: 15 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:50,083 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531929558_0001_00_000002, Exiting
2016-05-18 00:38:50,083 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531929558_0001_00_000002
2016-05-18 00:38:50,083 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531929558_0001_00_000002
2016-05-18 00:38:50,084 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531929558_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:50,084 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531929558_0001_00_000002 completed successfully
2016-05-18 00:38:50,085 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463531929558_0001_00_000002
2016-05-18 00:38:50,088 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:86A9A9EC012D4868BB1F5EDFD40A2A9E of vertex/vertexGroup:A680905727274BA4B95D68B330AC2818 isVertexGroupOutput:false
2016-05-18 00:38:50,088 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:50,092 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531929558_0001_1, queueSize=0
2016-05-18 00:38:50,094 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531929558_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531929558_0001_1, startTime=1463531929746, finishTime=1463531930088, timeTaken=342, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=650, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063860010, Process_Duration=12, Process_End_Time=2927063860022, Read_Duration=1, Tuples_Read=18, Tuples_Written=18, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 00:38:50,095 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531929558_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:50,095 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531929558_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:50 Completed Dag: dag_1463531929558_0001_1
2016-05-18 00:38:50,095 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531929558_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:50,096 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531929558_0001_1_post
2016-05-18 00:38:50,096 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,096 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:50,096 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:50,096 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn, with id=dag_1463531929558_0001_1
2016-05-18 00:38:50,096 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483] test/(1/1) ...irstNBufferForward/firstn, with id=dag_1463531929558_0001_1
2016-05-18 00:38:50,106 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463531929558_0001
2016-05-18 00:38:50,113 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:50,114 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:50,114 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:50,114 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:50,114 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:50,114 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:50,116 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:50,117 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:50,118 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:50,118 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:50,120 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:50,121 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:50,122 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:50,124 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:50,129 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/68C8B11EF0ED446DBC2724A7461CA5A9/FD7A38A1F4664B52893EB6D2F88FD483/.tez/application_1463531929558_0001
2016-05-18 00:38:50,130 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn/_temporary
2016-05-18 00:38:50,131 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.603
2016-05-18 00:38:50,233 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-18 00:38:50,268 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:50,269 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:50,359 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.030
2016-05-18 00:38:50,396 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.067
2016-05-18 00:38:50,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:50,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:50,398 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:50,418 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:50,427 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:50,498 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:50,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testSimpleBuffer/simple
2016-05-18 00:38:50,500 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:50,516 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:50,521 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:50,521 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:50,522 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:50,531 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001 doesn't exist and is created
2016-05-18 00:38:50,551 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:50,552 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:50,553 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd
2016-05-18 00:38:50,568 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531930521_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:50,569 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:50,569 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:50,571 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:50,572 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:50,572 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:50,572 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:50,575 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:50,576 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:50,577 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:50,577 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:50,578 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:50,578 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:50,578 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:50,578 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:50,580 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:50,582 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:50,582 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:50,582 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-log-dir
2016-05-18 00:38:50,583 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-log-dir/history.txt.appattempt_1463531930521_0001_000000, maxErrors=10
2016-05-18 00:38:50,583 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:50,583 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531930521_0001_000000, appSubmitTime=1463531930568, launchTime=1463531930568
2016-05-18 00:38:50,591 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:50,594 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:37477
2016-05-18 00:38:50,596 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531930521
2016-05-18 00:38:50,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531930521_0001_000000, startTime=1463531930598
2016-05-18 00:38:50,598 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:50,652 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:50,652 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:50,652 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463531930521_0001, dagName=[1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 00:38:50,673 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 00:38:50,674 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:50,675 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531930521_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-log-dir/dag_1463531930521_0001_1.dot
2016-05-18 00:38:50,676 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-log-dir/dag_1463531930521_0001_1-tez-dag.pb.txt
2016-05-18 00:38:50,684 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531930521_0001_1
2016-05-18 00:38:50,684 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,684 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 00:38:50 Running Dag: dag_1463531930521_0001_1
2016-05-18 00:38:50,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531930521_0001_1, submitTime=1463531930673
2016-05-18 00:38:50,696 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001/tez-conf.pb" } size: 36903 timestamp: 1463531930000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:50,696 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,696 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:50,697 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001/tez-conf.pb" } size: 36903 timestamp: 1463531930000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:50,697 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,698 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:50,699 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:50,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531930521_0001_1, initTime=1463531930699
2016-05-18 00:38:50,699 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531930521_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:50,700 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:50,701 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:DAG_STARTED]: dagID=dag_1463531930521_0001_1, startTime=1463531930700
2016-05-18 00:38:50,701 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531930521_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:50,701 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C6361C8B4B14455D8FCE5A44E4140AEF : {9B0A90996FD847A49A68931E13B3C3E6={InputName=9B0A90996FD847A49A68931E13B3C3E6}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:50,701 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 9B0A90996FD847A49A68931E13B3C3E6, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:50,701 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463531930521_0001, dagName=[1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 00:38:50,701 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,702 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,702 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,702 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,703 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:50,703 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,705 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 9B0A90996FD847A49A68931E13B3C3E6 on vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,706 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:50,706 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,707 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531930521_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:50,708 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9], numAdditionalOutputs=1
2016-05-18 00:38:50,709 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=DEB1D2E02ECF4C108E2C4897FCA32D3B, vertex=vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:50,709 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 9B0A90996FD847A49A68931E13B3C3E6 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:50,711 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for ED4404FF3E8445C2B248D498F26C9BF9:DEB1D2E02ECF4C108E2C4897FCA32D3B using oldmapred API
2016-05-18 00:38:50,712 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=DEB1D2E02ECF4C108E2C4897FCA32D3B, vertexName=ED4404FF3E8445C2B248D498F26C9BF9, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:50,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, vertexId=vertex_1463531930521_0001_1_01, initRequestedTime=1463531930703, initedTime=1463531930708, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:50,714 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:50,714 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:50,715 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:50,715 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:50,714 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:50,715 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 9B0A90996FD847A49A68931E13B3C3E6 on vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,716 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:50,717 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {9B0A90996FD847A49A68931E13B3C3E6=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:50,717 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] parallelism set to 1
2016-05-18 00:38:50,717 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:50,718 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, vertexId=vertex_1463531930521_0001_1_00, initRequestedTime=1463531930701, initedTime=1463531930718, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:50,718 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] with distanceFromRoot: 0
2016-05-18 00:38:50,718 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:50,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531930521_0001_1_00, startRequestedTime=1463531930703, startedTime=1463531930719
2016-05-18 00:38:50,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531930521_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:50,720 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:50,720 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531930521_0001_1_00 for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] numStartedSources: 1 numSources: 1
2016-05-18 00:38:50,720 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C6361C8B4B14455D8FCE5A44E4140AEF
2016-05-18 00:38:50,720 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,720 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] with distanceFromRoot: 1
2016-05-18 00:38:50,722 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531930521_0001_1_01, startRequestedTime=1463531930720, startedTime=1463531930721
2016-05-18 00:38:50,722 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:50,723 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: ED4404FF3E8445C2B248D498F26C9BF9 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:50,723 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_STARTED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, taskId=task_1463531930521_0001_1_00_000000, scheduledTime=1463531930722, launchTime=1463531930722
2016-05-18 00:38:50,724 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C6361C8B4B14455D8FCE5A44E4140AEF in vertex: ED4404FF3E8445C2B248D498F26C9BF9 numBipartiteSourceTasks: 1
2016-05-18 00:38:50,726 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,728 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:50,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531930521_0001_00_000001, launchTime=1463531930730
2016-05-18 00:38:50,730 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531930521_0001_00_000001
2016-05-18 00:38:50,731 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531930521_0001_1_00_000000_0] started. Is using containerId: [container_1463531930521_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:50,732 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531930521_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0, startTime=1463531930731, containerId=container_1463531930521_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:50 Starting to run new task attempt: attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,733 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,733 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,733 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:50,734 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: C6361C8B4B14455D8FCE5A44E4140AEF, VertexParallelism: 1, TaskAttemptID:attempt_1463531930521_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9B0A90996FD847A49A68931E13B3C3E6, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=ED4404FF3E8445C2B248D498F26C9BF9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:50,734 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:50,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531930521_0001, with dag id: dag_1463531930521_0001_1
2016-05-18 00:38:50,737 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531930521_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:50,738 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531930521_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:50,738 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,745 INFO input.MRInput (MRInput.java:initialize(440)) - 9B0A90996FD847A49A68931E13B3C3E6 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:50,745 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 9B0A90996FD847A49A68931E13B3C3E6
2016-05-18 00:38:50,746 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:50,754 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C6361C8B4B14455D8FCE5A44E4140AEF, ordinal: 0
2016-05-18 00:38:50,754 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6361C8B4B14455D8FCE5A44E4140AEF, mem on start (mb), free: 181, total: 490, max: 672
2016-05-18 00:38:50,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:50,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:50,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:50,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:50,755 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[9B0A90996FD847A49A68931E13B3C3E6:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [ED4404FF3E8445C2B248D498F26C9BF9:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:50,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:50,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:50,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:50,755 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[9B0A90996FD847A49A68931E13B3C3E6:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [ED4404FF3E8445C2B248D498F26C9BF9:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:50,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:50,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 9B0A90996FD847A49A68931E13B3C3E6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:50,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:50,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:50,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 9B0A90996FD847A49A68931E13B3C3E6
2016-05-18 00:38:50,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:50,756 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,758 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:50,759 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 9B0A90996FD847A49A68931E13B3C3E6 initialized RecordReader from event
2016-05-18 00:38:50,761 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:50,762 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 9B0A90996FD847A49A68931E13B3C3E6
2016-05-18 00:38:50,762 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: C0E5D04167E34173B11A047B5361891A
2016-05-18 00:38:50,762 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] C0E5D04167E34173B11A047B5361891A
2016-05-18 00:38:50,764 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - ED4404FF3E8445C2B248D498F26C9BF9 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6d3deca1, comparator=cascading.tuple.hadoop.util.TupleComparator@7c7a760e, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:50,764 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for ED4404FF3E8445C2B248D498F26C9BF9: , UsingHashComparator=false
2016-05-18 00:38:50,825 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:50,826 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:50,826 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for ED4404FF3E8445C2B248D498F26C9BF9: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:50,826 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - ED4404FF3E8445C2B248D498F26C9BF9: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:50,827 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:50,828 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C6361C8B4B14455D8FCE5A44E4140AEF, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:50,830 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,831 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C6361C8B4B14455D8FCE5A44E4140AEF, mem on close (mb), free: 79, total: 490, max: 672
2016-05-18 00:38:50,831 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - ED4404FF3E8445C2B248D498F26C9BF9: Starting flush of map output
2016-05-18 00:38:50,831 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - ED4404FF3E8445C2B248D498F26C9BF9: Span0.length = 13, perItem = 15
2016-05-18 00:38:50,832 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - ED4404FF3E8445C2B248D498F26C9BF9: done sorting span=0, length=13, time=0
2016-05-18 00:38:50,832 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - ED4404FF3E8445C2B248D498F26C9BF9: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 00:38:50,834 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - ED4404FF3E8445C2B248D498F26C9BF9: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-local-dir/output/attempt_1463531930521_0001_1_00_000000_0_10015_0/file.out
2016-05-18 00:38:50,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:50 Completed running task attempt: attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,838 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:50,838 WARN task.TaskReporter (TaskReporter.java:call(214)) - Exiting TaskReporter thread with pending queue size=2
2016-05-18 00:38:50,839 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531930521_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:50,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531930521_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531930828, Process_Duration=2, Process_End_Time=1463531930830, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:50,842 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531930521_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,842 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,842 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:50,842 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,843 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, taskAttemptId=attempt_1463531930521_0001_1_00_000000_0, creationTime=1463531930724, allocationTime=1463531930729, startTime=1463531930731, finishTime=1463531930843, timeTaken=112, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:50,844 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531930521_0001_00_000001
2016-05-18 00:38:50,844 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531930521_0001_1_00_000000_0
2016-05-18 00:38:50,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_FINISHED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, taskId=task_1463531930521_0001_1_00_000000, startTime=1463531930731, finishTime=1463531930845, timeTaken=114, status=SUCCEEDED, successfulAttemptID=attempt_1463531930521_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531930828, Process_Duration=2, Process_End_Time=1463531930830, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:50,846 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:50,846 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,848 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_FINISHED]: vertexName=C6361C8B4B14455D8FCE5A44E4140AEF, vertexId=vertex_1463531930521_0001_1_00, initRequestedTime=1463531930701, initedTime=1463531930718, startRequestedTime=1463531930703, startedTime=1463531930719, finishTime=1463531930847, timeTaken=128, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531930828, Process_Duration=2, Process_End_Time=1463531930830, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531930731, firstTasksToStart=[ task_1463531930521_0001_1_00_000000 ], lastTaskFinishTime=1463531930843, lastTasksToFinish=[ task_1463531930521_0001_1_00_000000 ], minTaskDuration=112, maxTaskDuration=112, avgTaskDuration=112.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531930521_0001_1_00_000000 ], longestDurationTasks=[ task_1463531930521_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:50,848 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:50,848 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531930521_0001_00_000001, stoppedTime=1463531930848, exitStatus=0
2016-05-18 00:38:50,848 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] attempt: attempt_1463531930521_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:50,848 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:50,849 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:50,849 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ED4404FF3E8445C2B248D498F26C9BF9
2016-05-18 00:38:50,849 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531930521_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:50,849 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C6361C8B4B14455D8FCE5A44E4140AEF, destinationVertexName=ED4404FF3E8445C2B248D498F26C9BF9. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:50,850 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_STARTED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, taskId=task_1463531930521_0001_1_01_000000, scheduledTime=1463531930850, launchTime=1463531930850
2016-05-18 00:38:50,851 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,853 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531930521_0001_00_000002, launchTime=1463531930852
2016-05-18 00:38:50,853 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531930521_0001_00_000002
2016-05-18 00:38:50,854 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531930521_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,854 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531930521_0001_1_01_000000_0] started. Is using containerId: [container_1463531930521_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:50,854 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0, startTime=1463531930854, containerId=container_1463531930521_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:50 Starting to run new task attempt: attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,855 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,855 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,855 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:50,855 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531930521_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:50,856 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: ED4404FF3E8445C2B248D498F26C9BF9, VertexParallelism: 1, TaskAttemptID:attempt_1463531930521_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C6361C8B4B14455D8FCE5A44E4140AEF, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DEB1D2E02ECF4C108E2C4897FCA32D3B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:50,856 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:50,856 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531930521_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:50,858 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531930521_0001_00_000001, Exiting
2016-05-18 00:38:50,858 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531930521_0001_00_000001
2016-05-18 00:38:50,858 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,859 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531930521_0001_00_000001
2016-05-18 00:38:50,861 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531930521_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,862 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531930521_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:50,865 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531930521_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:50,865 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531930521_0001_00_000001 completed successfully
2016-05-18 00:38:50,867 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:50,868 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:50,879 INFO output.MROutput (MROutput.java:initialize(421)) - DEB1D2E02ECF4C108E2C4897FCA32D3B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:50,881 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ED4404FF3E8445C2B248D498F26C9BF9, ordinal: 1
2016-05-18 00:38:50,881 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED4404FF3E8445C2B248D498F26C9BF9, mem on start (mb), free: 73, total: 490, max: 672
2016-05-18 00:38:50,881 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:50,881 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:50,882 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:50,882 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:50,882 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DEB1D2E02ECF4C108E2C4897FCA32D3B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C6361C8B4B14455D8FCE5A44E4140AEF:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:38:50,882 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:50,882 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:50,883 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:50,883 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DEB1D2E02ECF4C108E2C4897FCA32D3B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C6361C8B4B14455D8FCE5A44E4140AEF:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:38:50,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:50,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C6361C8B4B14455D8FCE5A44E4140AEF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:50,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:50,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:50,884 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C6361C8B4B14455D8FCE5A44E4140AEF: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:50,884 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C6361C8B4B14455D8FCE5A44E4140AEF: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:50,885 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C6361C8B4B14455D8FCE5A44E4140AEF with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:50,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C6361C8B4B14455D8FCE5A44E4140AEF
2016-05-18 00:38:50,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:50,888 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,889 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C6361C8B4B14455D8FCE5A44E4140AEF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:50,893 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531930521_0001_1_00_000000_0_10015} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463531930893, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:38:50,893 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C6361C8B4B14455D8FCE5A44E4140AEF
2016-05-18 00:38:50,893 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:50,895 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C6361C8B4B14455D8FCE5A44E4140AEF, wasInterrupted=false
2016-05-18 00:38:50,896 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - C6361C8B4B14455D8FCE5A44E4140AEF: Already shutdown. Ignoring fetch complete
2016-05-18 00:38:50,896 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:50,898 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:50,899 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:50,899 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001_wd/localmode-local-dir/output/attempt_1463531930521_0001_1_00_000000_0_10015/file.out, len=175, isLocal=true
2016-05-18 00:38:50,899 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 00:38:50,900 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:50,900 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:50,901 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 00:38:50,901 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C6361C8B4B14455D8FCE5A44E4140AEF
2016-05-18 00:38:50,901 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: C0E5D04167E34173B11A047B5361891A
2016-05-18 00:38:50,904 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"], id: DEB1D2E02ECF4C108E2C4897FCA32D3B
2016-05-18 00:38:50,905 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C6361C8B4B14455D8FCE5A44E4140AEF: Shuffle Runner thread complete
2016-05-18 00:38:50,904 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-18 00:38:50,906 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] C0E5D04167E34173B11A047B5361891A, for 1 inputs
2016-05-18 00:38:50,906 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ED4404FF3E8445C2B248D498F26C9BF9, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:50,907 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C6361C8B4B14455D8FCE5A44E4140AEF: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:50,912 INFO output.MROutput (MROutput.java:close(526)) - DEB1D2E02ECF4C108E2C4897FCA32D3B closed
2016-05-18 00:38:50,913 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531930521_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:50,914 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,914 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED4404FF3E8445C2B248D498F26C9BF9, mem on close (mb), free: 66, total: 490, max: 672
2016-05-18 00:38:50,915 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C6361C8B4B14455D8FCE5A44E4140AEF
2016-05-18 00:38:50,916 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C6361C8B4B14455D8FCE5A44E4140AEF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:50,916 INFO output.MROutput (MROutput.java:close(526)) - DEB1D2E02ECF4C108E2C4897FCA32D3B closed
2016-05-18 00:38:50,917 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:50 Completed running task attempt: attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:50,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531930521_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:50,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531930521_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531930907, Process_Duration=7, Process_End_Time=1463531930914, Read_Duration=0, Tuples_Read=13, Tuples_Written=23, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=23]]
2016-05-18 00:38:50,921 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531930521_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,921 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, taskAttemptId=attempt_1463531930521_0001_1_01_000000_0, creationTime=1463531930851, allocationTime=1463531930852, startTime=1463531930854, finishTime=1463531930921, timeTaken=67, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:50,922 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531930521_0001_1_01_000000_0
2016-05-18 00:38:50,921 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:50,923 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:50,923 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:TASK_FINISHED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, taskId=task_1463531930521_0001_1_01_000000, startTime=1463531930854, finishTime=1463531930922, timeTaken=68, status=SUCCEEDED, successfulAttemptID=attempt_1463531930521_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531930907, Process_Duration=7, Process_End_Time=1463531930914, Read_Duration=0, Tuples_Read=13, Tuples_Written=23, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=23
2016-05-18 00:38:50,924 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:50,924 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9]
2016-05-18 00:38:50,924 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531930521_0001_00_000002
2016-05-18 00:38:50,925 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:VERTEX_FINISHED]: vertexName=ED4404FF3E8445C2B248D498F26C9BF9, vertexId=vertex_1463531930521_0001_1_01, initRequestedTime=1463531930703, initedTime=1463531930708, startRequestedTime=1463531930720, startedTime=1463531930721, finishTime=1463531930924, timeTaken=203, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531930907, Process_Duration=7, Process_End_Time=1463531930914, Read_Duration=0, Tuples_Read=13, Tuples_Written=23, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=23, vertexStats=firstTaskStartTime=1463531930854, firstTasksToStart=[ task_1463531930521_0001_1_01_000000 ], lastTaskFinishTime=1463531930921, lastTasksToFinish=[ task_1463531930521_0001_1_01_000000 ], minTaskDuration=67, maxTaskDuration=67, avgTaskDuration=67.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531930521_0001_1_01_000000 ], longestDurationTasks=[ task_1463531930521_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:50,926 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:50,926 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531930521_0001_00_000002, stoppedTime=1463531930926, exitStatus=0
2016-05-18 00:38:50,926 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:50,927 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:50,927 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531930521_0001_1_00 [C6361C8B4B14455D8FCE5A44E4140AEF]
2016-05-18 00:38:50,927 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531930521_0001_1
2016-05-18 00:38:50,928 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531930521_0001_1
2016-05-18 00:38:50,928 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: DEB1D2E02ECF4C108E2C4897FCA32D3B for vertex: vertex_1463531930521_0001_1_01 [ED4404FF3E8445C2B248D498F26C9BF9], outputName: DEB1D2E02ECF4C108E2C4897FCA32D3B
2016-05-18 00:38:50,928 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531930521_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:50,929 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531930521_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:50,935 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531930521_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:50,935 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531930521_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:50,937 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531930521_0001_00_000002, Exiting
2016-05-18 00:38:50,938 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531930521_0001_00_000002
2016-05-18 00:38:50,938 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531930521_0001_00_000002
2016-05-18 00:38:50,939 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:DEB1D2E02ECF4C108E2C4897FCA32D3B of vertex/vertexGroup:ED4404FF3E8445C2B248D498F26C9BF9 isVertexGroupOutput:false
2016-05-18 00:38:50,939 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:50,939 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531930521_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:50,940 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531930521_0001_00_000002 completed successfully
2016-05-18 00:38:50,942 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531930521_0001_1, queueSize=0
2016-05-18 00:38:50,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531930521_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531930521_0001_1, startTime=1463531930700, finishTime=1463531930939, timeTaken=239, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=36, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063861735, Process_Duration=9, Process_End_Time=2927063861744, Read_Duration=1, Tuples_Read=26, Tuples_Written=36, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=23
2016-05-18 00:38:50,942 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531930521_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:50,943 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531930521_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:50 Completed Dag: dag_1463531930521_0001_1
2016-05-18 00:38:50,943 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531930521_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:50,943 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531930521_0001_1_post
2016-05-18 00:38:50,944 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:50,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:50,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:50,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple, with id=dag_1463531930521_0001_1
2016-05-18 00:38:50,945 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264] test/(1/1) ...m/testSimpleBuffer/simple, with id=dag_1463531930521_0001_1
2016-05-18 00:38:50,951 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463531930521_0001
2016-05-18 00:38:50,955 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:50,955 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:50,955 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:50,955 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:50,956 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:50,956 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:50,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:50,957 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:50,958 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:50,958 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:50,958 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:50,959 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:50,959 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:50,961 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:50,965 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1DA21BA7428040B4A87EBF98BAEAD74D/F7FFD22B3D374995BA0BEB35DB7F5264/.tez/application_1463531930521_0001
2016-05-18 00:38:50,966 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple/_temporary
2016-05-18 00:38:50,967 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.468
2016-05-18 00:38:51,090 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-18 00:38:51,120 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:51,120 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:51,187 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.034
2016-05-18 00:38:51,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.060
2016-05-18 00:38:51,214 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 00:38:51,214 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 00:38:51,215 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:51,237 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt
2016-05-18 00:38:51,250 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt
2016-05-18 00:38:51,265 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-18 00:38:51,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting
2016-05-18 00:38:51,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 00:38:51,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 00:38:51,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-18 00:38:51,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] parallel execution of steps is enabled: false
2016-05-18 00:38:51,389 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executing total steps: 1
2016-05-18 00:38:51,389 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] allocating management threads: 1
2016-05-18 00:38:51,389 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting step: (1/1) ...testJoinerClosure/cogroup
2016-05-18 00:38:51,390 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [lower*upper] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:51,409 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tez session mode enabled: true
2016-05-18 00:38:51,413 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:51,413 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:51,413 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:51,423 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001 doesn't exist and is created
2016-05-18 00:38:51,438 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:51,438 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:51,439 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd
2016-05-18 00:38:51,450 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531931413_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:51,450 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:51,450 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:51,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:51,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:51,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:51,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:51,454 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:51,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:51,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:51,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:51,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:51,455 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:51,455 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:51,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:51,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:51,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:51,456 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:51,457 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:51,457 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:51,459 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:51,460 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:51,460 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:51,461 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-log-dir
2016-05-18 00:38:51,461 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-log-dir/history.txt.appattempt_1463531931413_0001_000000, maxErrors=10
2016-05-18 00:38:51,461 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:51,462 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531931413_0001_000000, appSubmitTime=1463531931449, launchTime=1463531931449
2016-05-18 00:38:51,468 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:51,473 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:38698
2016-05-18 00:38:51,474 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531931413
2016-05-18 00:38:51,475 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531931413_0001_000000, startTime=1463531931475
2016-05-18 00:38:51,475 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:51,538 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:51,538 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:51,539 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463531931413_0001, dagName=[AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 00:38:51,558 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 00:38:51,559 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:51,559 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531931413_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-log-dir/dag_1463531931413_0001_1.dot
2016-05-18 00:38:51,561 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-log-dir/dag_1463531931413_0001_1-tez-dag.pb.txt
2016-05-18 00:38:51,572 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531931413_0001_1
2016-05-18 00:38:51,572 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,572 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 00:38:51 Running Dag: dag_1463531931413_0001_1
2016-05-18 00:38:51,582 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531931413_0001_1, submitTime=1463531931558
2016-05-18 00:38:51,583 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001/tez-conf.pb" } size: 36903 timestamp: 1463531931000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:51,583 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,583 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:51,584 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001/tez-conf.pb" } size: 36903 timestamp: 1463531931000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:51,584 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,585 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:51,586 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001/tez-conf.pb" } size: 36903 timestamp: 1463531931000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:51,586 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,586 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:51,586 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:51,587 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531931413_0001_1, initTime=1463531931587
2016-05-18 00:38:51,587 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531931413_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:51,589 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:51,590 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:DAG_STARTED]: dagID=dag_1463531931413_0001_1, startTime=1463531931589
2016-05-18 00:38:51,590 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531931413_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:51,590 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B7A13CDCC8FC4F30A65A8DE93158F5F4 : {4FB1F07DA67D40F0A9B322B2699DDA64={InputName=4FB1F07DA67D40F0A9B322B2699DDA64}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:51,590 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463531931413_0001, dagName=[AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 00:38:51,590 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 4FB1F07DA67D40F0A9B322B2699DDA64, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:51,590 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,591 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,591 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,591 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B0C092AE73604302A72942731D63AED7 : {0742D262D9D949789887BCEEB4412AA7={InputName=0742D262D9D949789887BCEEB4412AA7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 0742D262D9D949789887BCEEB4412AA7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,592 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,593 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,594 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:51,594 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 0742D262D9D949789887BCEEB4412AA7 on vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,594 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,596 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 0742D262D9D949789887BCEEB4412AA7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:51,598 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:51,598 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,598 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531931413_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=2, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:51,598 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565], numAdditionalOutputs=1
2016-05-18 00:38:51,598 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=D2DF5A4770A44942A1C9786A0C6E1890, vertex=vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:51,601 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:51,601 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 19 Min splitLength: 52428800 New desired splits: 1 Total length: 19 Original splits: 3
2016-05-18 00:38:51,601 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 19 numLocations: 1 numSplitsPerLocation: 3 numSplitsInGroup: 3 totalLength: 19 numOriginalSplits: 3 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:51,601 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 3
2016-05-18 00:38:51,601 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 4FB1F07DA67D40F0A9B322B2699DDA64 on vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,602 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 0742D262D9D949789887BCEEB4412AA7 on vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,603 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F3CC22F7C24F412DB389515A9D667565:D2DF5A4770A44942A1C9786A0C6E1890 using oldmapred API
2016-05-18 00:38:51,603 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=D2DF5A4770A44942A1C9786A0C6E1890, vertexName=F3CC22F7C24F412DB389515A9D667565, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:51,605 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F3CC22F7C24F412DB389515A9D667565, vertexId=vertex_1463531931413_0001_1_02, initRequestedTime=1463531931594, initedTime=1463531931598, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:51,605 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:51,609 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:51,610 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 4FB1F07DA67D40F0A9B322B2699DDA64 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:51,610 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {0742D262D9D949789887BCEEB4412AA7=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:51,611 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] parallelism set to 1
2016-05-18 00:38:51,612 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:51,612 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B0C092AE73604302A72942731D63AED7, vertexId=vertex_1463531931413_0001_1_00, initRequestedTime=1463531931592, initedTime=1463531931612, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:51,612 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] with distanceFromRoot: 0
2016-05-18 00:38:51,612 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:51,612 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531931413_0001_1_00, startRequestedTime=1463531931594, startedTime=1463531931612
2016-05-18 00:38:51,613 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531931413_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:51,613 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:51,613 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531931413_0001_1_00 for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] numStartedSources: 1 numSources: 2
2016-05-18 00:38:51,613 INFO impl.VertexImpl (VertexImpl.java:transition(2937)) - Cannot start vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] numStartedSources: 1 numSources: 2
2016-05-18 00:38:51,614 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:51,615 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 19 Min splitLength: 52428800 New desired splits: 1 Total length: 19 Original splits: 3
2016-05-18 00:38:51,615 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 19 numLocations: 1 numSplitsPerLocation: 3 numSplitsInGroup: 3 totalLength: 19 numOriginalSplits: 3 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:51,615 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 3
2016-05-18 00:38:51,616 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B0C092AE73604302A72942731D63AED7
2016-05-18 00:38:51,616 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,616 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 4FB1F07DA67D40F0A9B322B2699DDA64 on vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,617 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_STARTED]: vertexName=B0C092AE73604302A72942731D63AED7, taskId=task_1463531931413_0001_1_00_000000, scheduledTime=1463531931616, launchTime=1463531931616
2016-05-18 00:38:51,617 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,617 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:51,618 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {4FB1F07DA67D40F0A9B322B2699DDA64=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:51,618 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:51,619 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] parallelism set to 1
2016-05-18 00:38:51,620 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:51,621 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, vertexId=vertex_1463531931413_0001_1_01, initRequestedTime=1463531931590, initedTime=1463531931620, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:51,621 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] with distanceFromRoot: 0
2016-05-18 00:38:51,621 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:51,621 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531931413_0001_1_01, startRequestedTime=1463531931594, startedTime=1463531931621
2016-05-18 00:38:51,621 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531931413_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:51,622 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:51,622 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531931413_0001_1_01 for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] numStartedSources: 2 numSources: 2
2016-05-18 00:38:51,622 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] with distanceFromRoot: 1
2016-05-18 00:38:51,622 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B7A13CDCC8FC4F30A65A8DE93158F5F4
2016-05-18 00:38:51,622 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531931413_0001_1_02, startRequestedTime=1463531931622, startedTime=1463531931622
2016-05-18 00:38:51,622 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531931413_0001_00_000001, launchTime=1463531931620
2016-05-18 00:38:51,622 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:51,622 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,625 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F3CC22F7C24F412DB389515A9D667565 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:51,625 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_STARTED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, taskId=task_1463531931413_0001_1_01_000000, scheduledTime=1463531931625, launchTime=1463531931625
2016-05-18 00:38:51,625 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,627 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B7A13CDCC8FC4F30A65A8DE93158F5F4 in vertex: F3CC22F7C24F412DB389515A9D667565 numBipartiteSourceTasks: 1
2016-05-18 00:38:51,628 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B0C092AE73604302A72942731D63AED7 in vertex: F3CC22F7C24F412DB389515A9D667565 numBipartiteSourceTasks: 2
2016-05-18 00:38:51,629 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000001
2016-05-18 00:38:51,631 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531931413_0001_00_000002, launchTime=1463531931631
2016-05-18 00:38:51,631 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531931413_0001_1_00_000000_0] started. Is using containerId: [container_1463531931413_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:51,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B0C092AE73604302A72942731D63AED7, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0, startTime=1463531931631, containerId=container_1463531931413_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:51,632 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000002
2016-05-18 00:38:51,634 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,634 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] submitted tez dag to app master: application_1463531931413_0001, with dag id: dag_1463531931413_0001_1
2016-05-18 00:38:51,634 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531931413_0001_1_01_000000_0] started. Is using containerId: [container_1463531931413_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:51 Starting to run new task attempt: attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,634 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,636 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,638 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0, startTime=1463531931633, containerId=container_1463531931413_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:51,638 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:51,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: B0C092AE73604302A72942731D63AED7, VertexParallelism: 1, TaskAttemptID:attempt_1463531931413_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=0742D262D9D949789887BCEEB4412AA7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F3CC22F7C24F412DB389515A9D667565, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:51,638 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:51 Starting to run new task attempt: attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,639 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,641 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,642 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,641 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531931413_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:51,643 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:51,643 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531931413_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:51,643 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: B7A13CDCC8FC4F30A65A8DE93158F5F4, VertexParallelism: 1, TaskAttemptID:attempt_1463531931413_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=4FB1F07DA67D40F0A9B322B2699DDA64, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F3CC22F7C24F412DB389515A9D667565, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:51,643 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:51,646 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,647 INFO input.MRInput (MRInput.java:initialize(440)) - 0742D262D9D949789887BCEEB4412AA7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:51,648 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 0742D262D9D949789887BCEEB4412AA7
2016-05-18 00:38:51,652 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531931413_0001_1_01_000000_0 sent events: (0-1).
2016-05-18 00:38:51,653 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:51,653 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531931413_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:51,660 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:51,662 INFO input.MRInput (MRInput.java:initialize(440)) - 4FB1F07DA67D40F0A9B322B2699DDA64 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:51,663 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 4FB1F07DA67D40F0A9B322B2699DDA64
2016-05-18 00:38:51,668 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B7A13CDCC8FC4F30A65A8DE93158F5F4, ordinal: 0
2016-05-18 00:38:51,668 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B7A13CDCC8FC4F30A65A8DE93158F5F4, mem on start (mb), free: 131, total: 484, max: 672
2016-05-18 00:38:51,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:51,668 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:51,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:51,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:51,669 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F3CC22F7C24F412DB389515A9D667565:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [4FB1F07DA67D40F0A9B322B2699DDA64:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:38:51,670 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:51,670 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:51,670 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:51,671 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F3CC22F7C24F412DB389515A9D667565:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [4FB1F07DA67D40F0A9B322B2699DDA64:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:38:51,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:51,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 4FB1F07DA67D40F0A9B322B2699DDA64 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:51,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:51,672 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:51,671 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B0C092AE73604302A72942731D63AED7, ordinal: 1
2016-05-18 00:38:51,672 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B0C092AE73604302A72942731D63AED7, mem on start (mb), free: 131, total: 484, max: 672
2016-05-18 00:38:51,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 4FB1F07DA67D40F0A9B322B2699DDA64
2016-05-18 00:38:51,673 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:51,673 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:51,674 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:51,674 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:51,674 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[0742D262D9D949789887BCEEB4412AA7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F3CC22F7C24F412DB389515A9D667565:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:51,673 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:51,675 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:51,676 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,680 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:51,680 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:51,680 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[0742D262D9D949789887BCEEB4412AA7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F3CC22F7C24F412DB389515A9D667565:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:51,680 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 00:38:51,681 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt:0+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt:9+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt:18+1], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=19}
2016-05-18 00:38:51,681 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 4FB1F07DA67D40F0A9B322B2699DDA64 initialized RecordReader from event
2016-05-18 00:38:51,682 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:51,682 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 0742D262D9D949789887BCEEB4412AA7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:51,683 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 0742D262D9D949789887BCEEB4412AA7
2016-05-18 00:38:51,683 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:51,684 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt"] streamed: true, id: 4FB1F07DA67D40F0A9B322B2699DDA64
2016-05-18 00:38:51,684 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: 7F8D0A9E5C3744C5AE3C6709FAEC1A48
2016-05-18 00:38:51,684 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 7F8D0A9E5C3744C5AE3C6709FAEC1A48
2016-05-18 00:38:51,684 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:51,686 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:51,686 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,687 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F3CC22F7C24F412DB389515A9D667565 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4ab6569e, comparator=cascading.tuple.hadoop.util.TupleComparator@772f2cd1, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:51,688 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F3CC22F7C24F412DB389515A9D667565: , UsingHashComparator=false
2016-05-18 00:38:51,711 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt:0+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt:9+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt:18+1], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=19}
2016-05-18 00:38:51,721 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 0742D262D9D949789887BCEEB4412AA7 initialized RecordReader from event
2016-05-18 00:38:51,720 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 00:38:51,720 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:51,723 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:51,723 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F3CC22F7C24F412DB389515A9D667565: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:51,723 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F3CC22F7C24F412DB389515A9D667565: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:51,723 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 00:38:51,724 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B7A13CDCC8FC4F30A65A8DE93158F5F4, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:38:51,723 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt"] streamed: true, id: 0742D262D9D949789887BCEEB4412AA7
2016-05-18 00:38:51,724 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: 7F8D0A9E5C3744C5AE3C6709FAEC1A48
2016-05-18 00:38:51,724 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 7F8D0A9E5C3744C5AE3C6709FAEC1A48
2016-05-18 00:38:51,727 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F3CC22F7C24F412DB389515A9D667565 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6de8c535, comparator=cascading.tuple.hadoop.util.TupleComparator@4ceb52f2, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:51,729 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F3CC22F7C24F412DB389515A9D667565: , UsingHashComparator=false
2016-05-18 00:38:51,731 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,752 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B7A13CDCC8FC4F30A65A8DE93158F5F4, mem on close (mb), free: 55, total: 490, max: 672
2016-05-18 00:38:51,752 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:51,753 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:51,753 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F3CC22F7C24F412DB389515A9D667565: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:51,754 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F3CC22F7C24F412DB389515A9D667565: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:51,754 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 00:38:51,754 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B0C092AE73604302A72942731D63AED7, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:51,753 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F3CC22F7C24F412DB389515A9D667565: Starting flush of map output
2016-05-18 00:38:51,756 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F3CC22F7C24F412DB389515A9D667565: Span0.length = 5, perItem = 15
2016-05-18 00:38:51,756 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F3CC22F7C24F412DB389515A9D667565: done sorting span=0, length=5, time=0
2016-05-18 00:38:51,757 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F3CC22F7C24F412DB389515A9D667565: Heap = SpanIterator<0:4> (span=Span[16777216,75]),
2016-05-18 00:38:51,758 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F3CC22F7C24F412DB389515A9D667565: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-local-dir/output/attempt_1463531931413_0001_1_01_000000_0_10023_0/file.out
2016-05-18 00:38:51,758 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,758 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B0C092AE73604302A72942731D63AED7, mem on close (mb), free: 54, total: 490, max: 672
2016-05-18 00:38:51,759 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F3CC22F7C24F412DB389515A9D667565: Starting flush of map output
2016-05-18 00:38:51,759 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F3CC22F7C24F412DB389515A9D667565: Span0.length = 5, perItem = 15
2016-05-18 00:38:51,759 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F3CC22F7C24F412DB389515A9D667565: done sorting span=0, length=5, time=0
2016-05-18 00:38:51,759 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F3CC22F7C24F412DB389515A9D667565: Heap = SpanIterator<0:4> (span=Span[16777216,75]),
2016-05-18 00:38:51,760 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F3CC22F7C24F412DB389515A9D667565: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-local-dir/output/attempt_1463531931413_0001_1_00_000000_0_10021_0/file.out
2016-05-18 00:38:51,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:51 Completed running task attempt: attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,763 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:51,763 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531931413_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:51,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531931413_0001_1_01_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531931724, Process_Duration=7, Process_End_Time=1463531931731, Read_Duration=6, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 00:38:51,766 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531931413_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,766 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,766 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, taskAttemptId=attempt_1463531931413_0001_1_01_000000_0, creationTime=1463531931625, allocationTime=1463531931629, startTime=1463531931633, finishTime=1463531931766, timeTaken=133, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:51,767 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531931413_0001_1_01_000000_0
2016-05-18 00:38:51,768 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_FINISHED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, taskId=task_1463531931413_0001_1_01_000000, startTime=1463531931633, finishTime=1463531931767, timeTaken=134, status=SUCCEEDED, successfulAttemptID=attempt_1463531931413_0001_1_01_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531931724, Process_Duration=7, Process_End_Time=1463531931731, Read_Duration=6, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 00:38:51,769 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:51,769 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,766 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:51,770 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,770 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000002
2016-05-18 00:38:51,771 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:51,772 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_FINISHED]: vertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, vertexId=vertex_1463531931413_0001_1_01, initRequestedTime=1463531931590, initedTime=1463531931620, startRequestedTime=1463531931594, startedTime=1463531931621, finishTime=1463531931769, timeTaken=148, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531931724, Process_Duration=7, Process_End_Time=1463531931731, Read_Duration=6, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463531931633, firstTasksToStart=[ task_1463531931413_0001_1_01_000000 ], lastTaskFinishTime=1463531931766, lastTasksToFinish=[ task_1463531931413_0001_1_01_000000 ], minTaskDuration=133, maxTaskDuration=133, avgTaskDuration=133.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531931413_0001_1_01_000000 ], longestDurationTasks=[ task_1463531931413_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:51,773 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:51,773 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] attempt: attempt_1463531931413_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:51,773 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531931413_0001_00_000002, stoppedTime=1463531931773, exitStatus=0
2016-05-18 00:38:51,774 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 00:38:51,774 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 00:38:51,774 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531931413_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:51 Completed running task attempt: attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,774 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:51,775 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531931413_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:51,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531931413_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531931755, Process_Duration=3, Process_End_Time=1463531931758, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 00:38:51,776 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531931413_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,777 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,777 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:51,777 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,777 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000001
2016-05-18 00:38:51,777 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B0C092AE73604302A72942731D63AED7, taskAttemptId=attempt_1463531931413_0001_1_00_000000_0, creationTime=1463531931617, allocationTime=1463531931620, startTime=1463531931631, finishTime=1463531931777, timeTaken=146, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:51,778 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531931413_0001_1_00_000000_0
2016-05-18 00:38:51,779 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_FINISHED]: vertexName=B0C092AE73604302A72942731D63AED7, taskId=task_1463531931413_0001_1_00_000000, startTime=1463531931631, finishTime=1463531931778, timeTaken=147, status=SUCCEEDED, successfulAttemptID=attempt_1463531931413_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531931755, Process_Duration=3, Process_End_Time=1463531931758, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 00:38:51,780 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:51,780 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_FINISHED]: vertexName=B0C092AE73604302A72942731D63AED7, vertexId=vertex_1463531931413_0001_1_00, initRequestedTime=1463531931592, initedTime=1463531931612, startRequestedTime=1463531931594, startedTime=1463531931612, finishTime=1463531931780, timeTaken=168, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531931755, Process_Duration=3, Process_End_Time=1463531931758, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463531931631, firstTasksToStart=[ task_1463531931413_0001_1_00_000000 ], lastTaskFinishTime=1463531931777, lastTasksToFinish=[ task_1463531931413_0001_1_00_000000 ], minTaskDuration=146, maxTaskDuration=146, avgTaskDuration=146.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531931413_0001_1_00_000000 ], longestDurationTasks=[ task_1463531931413_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:51,781 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:51,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531931413_0001_00_000001, stoppedTime=1463531931781, exitStatus=0
2016-05-18 00:38:51,781 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] attempt: attempt_1463531931413_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:51,782 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531931413_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:51,782 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:51,782 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531931413_0001_00_000002, Exiting
2016-05-18 00:38:51,782 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531931413_0001_00_000002
2016-05-18 00:38:51,783 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531931413_0001_00_000002
2016-05-18 00:38:51,783 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F3CC22F7C24F412DB389515A9D667565
2016-05-18 00:38:51,783 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B0C092AE73604302A72942731D63AED7, destinationVertexName=F3CC22F7C24F412DB389515A9D667565. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:51,783 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, destinationVertexName=F3CC22F7C24F412DB389515A9D667565. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:51,784 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,784 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531931413_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:51,784 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531931413_0001_00_000002 completed successfully
2016-05-18 00:38:51,784 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 00:38:51,784 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 00:38:51,784 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531931413_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:51,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_STARTED]: vertexName=F3CC22F7C24F412DB389515A9D667565, taskId=task_1463531931413_0001_1_02_000000, scheduledTime=1463531931785, launchTime=1463531931785
2016-05-18 00:38:51,785 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,789 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531931413_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:51,790 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:51,790 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531931413_0001_00_000001, Exiting
2016-05-18 00:38:51,790 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531931413_0001_00_000001
2016-05-18 00:38:51,790 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531931413_0001_00_000001
2016-05-18 00:38:51,791 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531931413_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:51,791 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531931413_0001_00_000001 completed successfully
2016-05-18 00:38:51,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531931413_0001_00_000003, launchTime=1463531931790
2016-05-18 00:38:51,792 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000003
2016-05-18 00:38:51,795 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,795 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531931413_0001_1_02_000000_0] started. Is using containerId: [container_1463531931413_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 00:38:51 Starting to run new task attempt: attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,795 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,796 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,796 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:51,795 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F3CC22F7C24F412DB389515A9D667565, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0, startTime=1463531931794, containerId=container_1463531931413_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 00:38:51,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: F3CC22F7C24F412DB389515A9D667565, VertexParallelism: 1, TaskAttemptID:attempt_1463531931413_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B0C092AE73604302A72942731D63AED7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=B7A13CDCC8FC4F30A65A8DE93158F5F4, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D2DF5A4770A44942A1C9786A0C6E1890, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:51,797 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=2, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:51,797 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,798 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531931413_0001_1_02_000000_0 2 events [0,2) total 2 vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,798 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531931413_0001_1_02_000000_0, eventCount=2 fromEventId=0 nextFromEventId=2
2016-05-18 00:38:51,800 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:51,801 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:51,803 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:51,813 INFO output.MROutput (MROutput.java:initialize(421)) - D2DF5A4770A44942A1C9786A0C6E1890: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:51,817 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F3CC22F7C24F412DB389515A9D667565, ordinal: 2
2016-05-18 00:38:51,817 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3CC22F7C24F412DB389515A9D667565, mem on start (mb), free: 45, total: 490, max: 672
2016-05-18 00:38:51,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:51,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 3 initializers to finish
2016-05-18 00:38:51,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:51,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:51,817 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:51,818 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B0C092AE73604302A72942731D63AED7:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [B7A13CDCC8FC4F30A65A8DE93158F5F4:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [D2DF5A4770A44942A1C9786A0C6E1890:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:38:51,819 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:51,819 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.045, finalReserveFractionUsed=0.345
2016-05-18 00:38:51,820 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 3, numScaledRequests: 25, TotalRequested: 422785824, TotalRequestedScaled: 2.0293719551999998E8, TotalJVMHeap: 704643072, TotalAvailable: 461541212, TotalRequested/TotalJVMHeap:0.60
2016-05-18 00:38:51,820 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B0C092AE73604302A72942731D63AED7:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [B7A13CDCC8FC4F30A65A8DE93158F5F4:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [D2DF5A4770A44942A1C9786A0C6E1890:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:38:51,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:51,821 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B0C092AE73604302A72942731D63AED7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:51,822 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B0C092AE73604302A72942731D63AED7: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:51,822 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B0C092AE73604302A72942731D63AED7: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:51,822 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B7A13CDCC8FC4F30A65A8DE93158F5F4 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:51,822 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:51,823 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:51,823 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 2
2016-05-18 00:38:51,823 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 2 IOs to start
2016-05-18 00:38:51,824 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B0C092AE73604302A72942731D63AED7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:51,824 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B7A13CDCC8FC4F30A65A8DE93158F5F4 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:51,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B7A13CDCC8FC4F30A65A8DE93158F5F4
2016-05-18 00:38:51,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:51,827 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B0C092AE73604302A72942731D63AED7
2016-05-18 00:38:51,827 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:51,827 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,832 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:51,832 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']]
2016-05-18 00:38:51,832 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B0C092AE73604302A72942731D63AED7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:51,833 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] streamed: true, id: 7F8D0A9E5C3744C5AE3C6709FAEC1A48
2016-05-18 00:38:51,834 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"], id: D2DF5A4770A44942A1C9786A0C6E1890
2016-05-18 00:38:51,834 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531931413_0001_1_00_000000_0_10021} to DISK_DIRECT, csize=95, dsize=91, EndTime=1463531931834, TimeTaken=1, Rate=0.09 MB/s
2016-05-18 00:38:51,834 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B0C092AE73604302A72942731D63AED7
2016-05-18 00:38:51,834 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-18 00:38:51,834 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:51,835 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B0C092AE73604302A72942731D63AED7, wasInterrupted=false
2016-05-18 00:38:51,835 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 7F8D0A9E5C3744C5AE3C6709FAEC1A48, for 2 inputs
2016-05-18 00:38:51,838 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:51,838 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 7F8D0A9E5C3744C5AE3C6709FAEC1A48, for 2 inputs
2016-05-18 00:38:51,839 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:51,852 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-local-dir/output/attempt_1463531931413_0001_1_00_000000_0_10021/file.out, len=95, isLocal=true
2016-05-18 00:38:51,852 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 95 bytes from disk
2016-05-18 00:38:51,852 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:51,852 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:51,853 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-18 00:38:51,853 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B0C092AE73604302A72942731D63AED7
2016-05-18 00:38:51,853 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B0C092AE73604302A72942731D63AED7: Shuffle Runner thread complete
2016-05-18 00:38:51,838 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531931413_0001_1_01_000000_0_10023} to DISK_DIRECT, csize=95, dsize=91, EndTime=1463531931838, TimeTaken=1, Rate=0.09 MB/s
2016-05-18 00:38:51,855 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B7A13CDCC8FC4F30A65A8DE93158F5F4
2016-05-18 00:38:51,856 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:51,856 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B7A13CDCC8FC4F30A65A8DE93158F5F4, wasInterrupted=false
2016-05-18 00:38:51,857 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:51,857 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:51,858 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001_wd/localmode-local-dir/output/attempt_1463531931413_0001_1_01_000000_0_10023/file.out, len=95, isLocal=true
2016-05-18 00:38:51,858 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 95 bytes from disk
2016-05-18 00:38:51,859 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:51,859 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:38:51,859 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:51,859 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 00:38:51,859 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-18 00:38:51,859 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B7A13CDCC8FC4F30A65A8DE93158F5F4
2016-05-18 00:38:51,860 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: Shuffle Runner thread complete
2016-05-18 00:38:51,864 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F3CC22F7C24F412DB389515A9D667565, all 2 inputs ready in: 00:00:00.000
2016-05-18 00:38:51,864 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:51,866 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B0C092AE73604302A72942731D63AED7: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:51,870 INFO output.MROutput (MROutput.java:close(526)) - D2DF5A4770A44942A1C9786A0C6E1890 closed
2016-05-18 00:38:51,871 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531931413_0001_1_02_000000_0 given a go for committing the task output.
2016-05-18 00:38:51,872 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,872 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3CC22F7C24F412DB389515A9D667565, mem on close (mb), free: 257, total: 490, max: 672
2016-05-18 00:38:51,872 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B0C092AE73604302A72942731D63AED7
2016-05-18 00:38:51,873 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B0C092AE73604302A72942731D63AED7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:51,873 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B7A13CDCC8FC4F30A65A8DE93158F5F4
2016-05-18 00:38:51,873 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B7A13CDCC8FC4F30A65A8DE93158F5F4: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:51,873 INFO output.MROutput (MROutput.java:close(526)) - D2DF5A4770A44942A1C9786A0C6E1890 closed
2016-05-18 00:38:51,873 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:51 Completed running task attempt: attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,874 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0, askedToStop=false
2016-05-18 00:38:51,874 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531931413_0001_1_02_000000_0, stopRequested=false
2016-05-18 00:38:51,875 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531931413_0001_1_02_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=49, MERGE_PHASE_TIME=67, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531931864, Process_Duration=8, Process_End_Time=1463531931872, Read_Duration=0, Tuples_Read=10, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 00:38:51,876 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531931413_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,876 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:51,876 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F3CC22F7C24F412DB389515A9D667565, taskAttemptId=attempt_1463531931413_0001_1_02_000000_0, creationTime=1463531931785, allocationTime=1463531931786, startTime=1463531931794, finishTime=1463531931876, timeTaken=82, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:51,877 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531931413_0001_1_02_000000_0
2016-05-18 00:38:51,877 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:51,878 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:TASK_FINISHED]: vertexName=F3CC22F7C24F412DB389515A9D667565, taskId=task_1463531931413_0001_1_02_000000, startTime=1463531931794, finishTime=1463531931877, timeTaken=83, status=SUCCEEDED, successfulAttemptID=attempt_1463531931413_0001_1_02_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=49, MERGE_PHASE_TIME=67, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531931864, Process_Duration=8, Process_End_Time=1463531931872, Read_Duration=0, Tuples_Read=10, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 00:38:51,878 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,879 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531931413_0001_00_000003
2016-05-18 00:38:51,878 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:51,880 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565]
2016-05-18 00:38:51,881 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:VERTEX_FINISHED]: vertexName=F3CC22F7C24F412DB389515A9D667565, vertexId=vertex_1463531931413_0001_1_02, initRequestedTime=1463531931594, initedTime=1463531931598, startRequestedTime=1463531931622, startedTime=1463531931622, finishTime=1463531931880, timeTaken=258, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=49, MERGE_PHASE_TIME=67, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531931864, Process_Duration=8, Process_End_Time=1463531931872, Read_Duration=0, Tuples_Read=10, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463531931794, firstTasksToStart=[ task_1463531931413_0001_1_02_000000 ], lastTaskFinishTime=1463531931876, lastTasksToFinish=[ task_1463531931413_0001_1_02_000000 ], minTaskDuration=82, maxTaskDuration=82, avgTaskDuration=82.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531931413_0001_1_02_000000 ], longestDurationTasks=[ task_1463531931413_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:51,881 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:51,881 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531931413_0001_00_000003, stoppedTime=1463531931881, exitStatus=0
2016-05-18 00:38:51,882 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 00:38:51,882 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 00:38:51,882 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531931413_0001_1_01 [B7A13CDCC8FC4F30A65A8DE93158F5F4]
2016-05-18 00:38:51,882 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531931413_0001_1_00 [B0C092AE73604302A72942731D63AED7]
2016-05-18 00:38:51,882 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531931413_0001_1
2016-05-18 00:38:51,882 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531931413_0001_1
2016-05-18 00:38:51,883 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: D2DF5A4770A44942A1C9786A0C6E1890 for vertex: vertex_1463531931413_0001_1_02 [F3CC22F7C24F412DB389515A9D667565], outputName: D2DF5A4770A44942A1C9786A0C6E1890
2016-05-18 00:38:51,883 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531931413_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:51,883 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531931413_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 00:38:51,889 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531931413_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 00:38:51,889 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531931413_0001_00_000003: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:51,889 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531931413_0001_00_000003, Exiting
2016-05-18 00:38:51,890 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531931413_0001_00_000003
2016-05-18 00:38:51,890 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531931413_0001_00_000003
2016-05-18 00:38:51,890 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531931413_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:51,890 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531931413_0001_00_000003 completed successfully
2016-05-18 00:38:51,890 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:D2DF5A4770A44942A1C9786A0C6E1890 of vertex/vertexGroup:F3CC22F7C24F412DB389515A9D667565 isVertexGroupOutput:false
2016-05-18 00:38:51,891 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 00:38:51,894 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531931413_0001_1, queueSize=0
2016-05-18 00:38:51,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531931413_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531931413_0001_1, startTime=1463531931589, finishTime=1463531931891, timeTaken=302, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=3, TOTAL_LAUNCHED_TASKS=3, RACK_LOCAL_TASKS=2, AM_CPU_MILLISECONDS=680, AM_GC_TIME_MILLIS=29, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=15, OUTPUT_BYTES=150, OUTPUT_BYTES_WITH_OVERHEAD=182, OUTPUT_BYTES_PHYSICAL=190, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=2, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=49, MERGE_PHASE_TIME=67, FIRST_EVENT_RECEIVED=19, LAST_EVENT_RECEIVED=19, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=4390595795343, Process_Duration=18, Process_End_Time=4390595795361, Read_Duration=8, Tuples_Read=20, Tuples_Written=15, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=10, Tuples_Written=5
2016-05-18 00:38:51,896 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531931413_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:51,896 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531931413_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:51 Completed Dag: dag_1463531931413_0001_1
2016-05-18 00:38:51,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531931413_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:51,898 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531931413_0001_1_post
2016-05-18 00:38:51,899 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:51,900 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:51,900 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:51,901 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup, with id=dag_1463531931413_0001_1
2016-05-18 00:38:51,901 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8] lower*upper/(1/1) ...testJoinerClosure/cogroup, with id=dag_1463531931413_0001_1
2016-05-18 00:38:51,904 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463531931413_0001
2016-05-18 00:38:51,907 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:51,907 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:51,907 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:51,907 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:51,907 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:51,908 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:51,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:51,909 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:51,909 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:51,909 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:51,910 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:51,911 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:51,911 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:51,912 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:51,915 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AFB3B50196C040FDB20C6FDA05145B52/A1A25A370AF64E59AACC89FDCD82E9C8/.tez/application_1463531931413_0001
2016-05-18 00:38:51,916 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup/_temporary
2016-05-18 00:38:51,916 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] completed in: 00:00.529
2016-05-18 00:38:52,002 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-18 00:38:52,032 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:52,032 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:52,087 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.023
2016-05-18 00:38:52,105 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.042
2016-05-18 00:38:52,105 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:52,106 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:52,106 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:52,133 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:52,145 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-18 00:38:52,212 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:52,212 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:52,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-18 00:38:52,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:52,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:52,214 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:52,215 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer2/simple2
2016-05-18 00:38:52,215 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:52,233 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:52,234 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:52,234 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:52,243 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001 doesn't exist and is created
2016-05-18 00:38:52,257 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:52,257 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:52,259 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd
2016-05-18 00:38:52,270 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531932233_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:52,271 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:52,271 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:52,272 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:52,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:52,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:52,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:52,274 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:52,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:52,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:52,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:52,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:52,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:52,276 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:52,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:52,277 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:52,277 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:52,277 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:52,277 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:52,277 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:52,279 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:52,280 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:52,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:52,280 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-log-dir
2016-05-18 00:38:52,280 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-log-dir/history.txt.appattempt_1463531932233_0001_000000, maxErrors=10
2016-05-18 00:38:52,281 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:52,281 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531932233_0001_000000, appSubmitTime=1463531932270, launchTime=1463531932270
2016-05-18 00:38:52,285 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:52,291 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:45552
2016-05-18 00:38:52,292 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531932233
2016-05-18 00:38:52,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531932233_0001_000000, startTime=1463531932294
2016-05-18 00:38:52,294 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:52,358 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:52,358 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:52,358 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463531932233_0001, dagName=[E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 00:38:52,376 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 00:38:52,377 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:52,377 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531932233_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-log-dir/dag_1463531932233_0001_1.dot
2016-05-18 00:38:52,379 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-log-dir/dag_1463531932233_0001_1-tez-dag.pb.txt
2016-05-18 00:38:52,389 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531932233_0001_1
2016-05-18 00:38:52,389 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,390 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 00:38:52 Running Dag: dag_1463531932233_0001_1
2016-05-18 00:38:52,399 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531932233_0001_1, submitTime=1463531932376
2016-05-18 00:38:52,400 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001/tez-conf.pb" } size: 36903 timestamp: 1463531932000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:52,400 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,400 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:52,401 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001/tez-conf.pb" } size: 36903 timestamp: 1463531932000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:52,401 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,401 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:52,401 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:52,402 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531932233_0001_1, initTime=1463531932401
2016-05-18 00:38:52,402 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932233_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:52,402 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:52,403 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:DAG_STARTED]: dagID=dag_1463531932233_0001_1, startTime=1463531932403
2016-05-18 00:38:52,403 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463531932233_0001, dagName=[E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 00:38:52,403 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932233_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:52,403 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B389936F7B9544EFBE78775452C4B1B7 : {B1F74DC2CA5C4EAC85ADD4BB3775FA6E={InputName=B1F74DC2CA5C4EAC85ADD4BB3775FA6E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:52,403 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: B1F74DC2CA5C4EAC85ADD4BB3775FA6E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:52,404 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,404 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,404 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,404 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,405 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:52,405 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,406 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: B1F74DC2CA5C4EAC85ADD4BB3775FA6E on vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,408 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:52,409 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,409 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,409 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input B1F74DC2CA5C4EAC85ADD4BB3775FA6E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:52,410 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531932233_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:52,411 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C], numAdditionalOutputs=1
2016-05-18 00:38:52,411 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=7F1E6629832F471589AF97168BEDF874, vertex=vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:52,413 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:52,413 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:52,413 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C:7F1E6629832F471589AF97168BEDF874 using oldmapred API
2016-05-18 00:38:52,414 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=7F1E6629832F471589AF97168BEDF874, vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:52,414 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:52,414 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:52,415 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: B1F74DC2CA5C4EAC85ADD4BB3775FA6E on vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,416 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, vertexId=vertex_1463531932233_0001_1_01, initRequestedTime=1463531932405, initedTime=1463531932411, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:52,416 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:52,417 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:52,418 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {B1F74DC2CA5C4EAC85ADD4BB3775FA6E=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:52,419 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] parallelism set to 1
2016-05-18 00:38:52,419 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:52,420 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, vertexId=vertex_1463531932233_0001_1_00, initRequestedTime=1463531932403, initedTime=1463531932419, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:52,420 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] with distanceFromRoot: 0
2016-05-18 00:38:52,420 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:52,421 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531932233_0001_1_00, startRequestedTime=1463531932405, startedTime=1463531932420
2016-05-18 00:38:52,421 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531932233_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:52,421 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:52,422 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531932233_0001_1_00 for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] numStartedSources: 1 numSources: 1
2016-05-18 00:38:52,422 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] with distanceFromRoot: 1
2016-05-18 00:38:52,422 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B389936F7B9544EFBE78775452C4B1B7
2016-05-18 00:38:52,423 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531932233_0001_1_01, startRequestedTime=1463531932422, startedTime=1463531932422
2016-05-18 00:38:52,423 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:52,423 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,425 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_STARTED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, taskId=task_1463531932233_0001_1_00_000000, scheduledTime=1463531932425, launchTime=1463531932425
2016-05-18 00:38:52,425 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,426 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C with 0 source tasks and 1 pending tasks
2016-05-18 00:38:52,426 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:52,428 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B389936F7B9544EFBE78775452C4B1B7 in vertex: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C numBipartiteSourceTasks: 1
2016-05-18 00:38:52,429 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531932233_0001_00_000001, launchTime=1463531932429
2016-05-18 00:38:52,430 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932233_0001_00_000001
2016-05-18 00:38:52,431 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531932233_0001_1_00_000000_0] started. Is using containerId: [container_1463531932233_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:52,431 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0, startTime=1463531932431, containerId=container_1463531932233_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:52,432 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531932233_0001, with dag id: dag_1463531932233_0001_1
2016-05-18 00:38:52,436 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932233_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52 Starting to run new task attempt: attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,436 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,437 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,437 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:52,437 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: B389936F7B9544EFBE78775452C4B1B7, VertexParallelism: 1, TaskAttemptID:attempt_1463531932233_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B1F74DC2CA5C4EAC85ADD4BB3775FA6E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:52,438 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:52,438 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,442 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531932233_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:52,442 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531932233_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:52,445 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:52,446 INFO input.MRInput (MRInput.java:initialize(440)) - B1F74DC2CA5C4EAC85ADD4BB3775FA6E using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:52,446 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: B1F74DC2CA5C4EAC85ADD4BB3775FA6E
2016-05-18 00:38:52,456 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B389936F7B9544EFBE78775452C4B1B7, ordinal: 0
2016-05-18 00:38:52,456 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B389936F7B9544EFBE78775452C4B1B7, mem on start (mb), free: 146, total: 490, max: 672
2016-05-18 00:38:52,457 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:52,457 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:52,457 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:52,458 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:52,458 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B1F74DC2CA5C4EAC85ADD4BB3775FA6E:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:52,459 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:52,460 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:52,460 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:52,461 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B1F74DC2CA5C4EAC85ADD4BB3775FA6E:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:52,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:52,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B1F74DC2CA5C4EAC85ADD4BB3775FA6E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:52,461 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B1F74DC2CA5C4EAC85ADD4BB3775FA6E
2016-05-18 00:38:52,462 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:52,462 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:52,463 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:52,463 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,465 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:52,466 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - B1F74DC2CA5C4EAC85ADD4BB3775FA6E initialized RecordReader from event
2016-05-18 00:38:52,467 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:52,468 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: B1F74DC2CA5C4EAC85ADD4BB3775FA6E
2016-05-18 00:38:52,468 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 3E0DFF35BF774131BDB989651DBD68DE
2016-05-18 00:38:52,468 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 3E0DFF35BF774131BDB989651DBD68DE
2016-05-18 00:38:52,470 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@418eabe8, comparator=cascading.tuple.hadoop.util.TupleComparator@7db82df7, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:52,470 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: , UsingHashComparator=false
2016-05-18 00:38:52,485 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:52,485 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:52,485 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:52,485 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:52,486 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:52,487 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B389936F7B9544EFBE78775452C4B1B7, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:52,489 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,490 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B389936F7B9544EFBE78775452C4B1B7, mem on close (mb), free: 45, total: 490, max: 672
2016-05-18 00:38:52,490 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: Starting flush of map output
2016-05-18 00:38:52,490 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: Span0.length = 13, perItem = 15
2016-05-18 00:38:52,491 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: done sorting span=0, length=13, time=0
2016-05-18 00:38:52,492 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 00:38:52,493 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-local-dir/output/attempt_1463531932233_0001_1_00_000000_0_10031_0/file.out
2016-05-18 00:38:52,496 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:52 Completed running task attempt: attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,497 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:52,497 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531932233_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:52,498 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531932233_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531932487, Process_Duration=2, Process_End_Time=1463531932489, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:52,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, taskAttemptId=attempt_1463531932233_0001_1_00_000000_0, creationTime=1463531932425, allocationTime=1463531932428, startTime=1463531932431, finishTime=1463531932499, timeTaken=68, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:52,500 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531932233_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:52,500 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:52,500 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:52,500 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,501 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932233_0001_00_000001
2016-05-18 00:38:52,501 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531932233_0001_1_00_000000_0
2016-05-18 00:38:52,502 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_FINISHED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, taskId=task_1463531932233_0001_1_00_000000, startTime=1463531932431, finishTime=1463531932501, timeTaken=70, status=SUCCEEDED, successfulAttemptID=attempt_1463531932233_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531932487, Process_Duration=2, Process_End_Time=1463531932489, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:52,502 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:52,503 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,504 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_FINISHED]: vertexName=B389936F7B9544EFBE78775452C4B1B7, vertexId=vertex_1463531932233_0001_1_00, initRequestedTime=1463531932403, initedTime=1463531932419, startRequestedTime=1463531932405, startedTime=1463531932420, finishTime=1463531932503, timeTaken=83, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531932487, Process_Duration=2, Process_End_Time=1463531932489, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531932431, firstTasksToStart=[ task_1463531932233_0001_1_00_000000 ], lastTaskFinishTime=1463531932499, lastTasksToFinish=[ task_1463531932233_0001_1_00_000000 ], minTaskDuration=68, maxTaskDuration=68, avgTaskDuration=68.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531932233_0001_1_00_000000 ], longestDurationTasks=[ task_1463531932233_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:52,504 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:52,504 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531932233_0001_00_000001, stoppedTime=1463531932504, exitStatus=0
2016-05-18 00:38:52,505 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] attempt: attempt_1463531932233_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:52,505 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:52,505 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C
2016-05-18 00:38:52,505 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:52,505 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B389936F7B9544EFBE78775452C4B1B7, destinationVertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:52,506 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,506 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531932233_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:52,506 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_STARTED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, taskId=task_1463531932233_0001_1_01_000000, scheduledTime=1463531932506, launchTime=1463531932506
2016-05-18 00:38:52,507 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531932233_0001_00_000002, launchTime=1463531932509
2016-05-18 00:38:52,510 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932233_0001_00_000002
2016-05-18 00:38:52,511 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531932233_0001_1_01_000000_0] started. Is using containerId: [container_1463531932233_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:52,511 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0, startTime=1463531932510, containerId=container_1463531932233_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:52,511 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932233_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,512 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531932233_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:52,512 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932233_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:52,512 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531932233_0001_00_000001, Exiting
2016-05-18 00:38:52,512 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531932233_0001_00_000001
2016-05-18 00:38:52,512 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531932233_0001_00_000001
2016-05-18 00:38:52 Starting to run new task attempt: attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,513 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,513 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,526 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:52,526 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, VertexParallelism: 1, TaskAttemptID:attempt_1463531932233_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B389936F7B9544EFBE78775452C4B1B7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=7F1E6629832F471589AF97168BEDF874, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:52,527 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:52,527 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531932233_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:52,528 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531932233_0001_00_000001 completed successfully
2016-05-18 00:38:52,528 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,531 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:52,533 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531932233_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,533 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531932233_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:52,533 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:52,544 INFO output.MROutput (MROutput.java:initialize(421)) - 7F1E6629832F471589AF97168BEDF874: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:52,544 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, ordinal: 1
2016-05-18 00:38:52,544 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, mem on start (mb), free: 254, total: 489, max: 672
2016-05-18 00:38:52,545 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:52,545 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:52,545 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:52,545 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:52,545 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B389936F7B9544EFBE78775452C4B1B7:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [7F1E6629832F471589AF97168BEDF874:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:38:52,546 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:52,546 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:52,546 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:52,547 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B389936F7B9544EFBE78775452C4B1B7:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [7F1E6629832F471589AF97168BEDF874:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:38:52,547 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:52,547 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B389936F7B9544EFBE78775452C4B1B7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:52,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:52,548 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B389936F7B9544EFBE78775452C4B1B7: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:52,548 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B389936F7B9544EFBE78775452C4B1B7: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:52,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:52,549 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B389936F7B9544EFBE78775452C4B1B7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:52,550 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B389936F7B9544EFBE78775452C4B1B7
2016-05-18 00:38:52,552 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:52,552 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,554 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B389936F7B9544EFBE78775452C4B1B7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:52,556 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:52,558 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 3E0DFF35BF774131BDB989651DBD68DE
2016-05-18 00:38:52,558 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"], id: 7F1E6629832F471589AF97168BEDF874
2016-05-18 00:38:52,558 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-18 00:38:52,559 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 3E0DFF35BF774131BDB989651DBD68DE, for 1 inputs
2016-05-18 00:38:52,559 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531932233_0001_1_00_000000_0_10031} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463531932559, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:38:52,559 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B389936F7B9544EFBE78775452C4B1B7
2016-05-18 00:38:52,559 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:52,560 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B389936F7B9544EFBE78775452C4B1B7, wasInterrupted=false
2016-05-18 00:38:52,561 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:52,561 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:52,562 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001_wd/localmode-local-dir/output/attempt_1463531932233_0001_1_00_000000_0_10031/file.out, len=175, isLocal=true
2016-05-18 00:38:52,562 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 00:38:52,562 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:52,562 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:52,563 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 00:38:52,563 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B389936F7B9544EFBE78775452C4B1B7
2016-05-18 00:38:52,563 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:38:52,563 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B389936F7B9544EFBE78775452C4B1B7: Shuffle Runner thread complete
2016-05-18 00:38:52,564 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B389936F7B9544EFBE78775452C4B1B7: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:52,569 INFO output.MROutput (MROutput.java:close(526)) - 7F1E6629832F471589AF97168BEDF874 closed
2016-05-18 00:38:52,569 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531932233_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:52,570 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,570 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, mem on close (mb), free: 251, total: 489, max: 672
2016-05-18 00:38:52,570 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B389936F7B9544EFBE78775452C4B1B7
2016-05-18 00:38:52,571 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B389936F7B9544EFBE78775452C4B1B7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:52,571 INFO output.MROutput (MROutput.java:close(526)) - 7F1E6629832F471589AF97168BEDF874 closed
2016-05-18 00:38:52,571 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:52 Completed running task attempt: attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,571 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:52,572 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531932233_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:52,572 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531932233_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531932563, Process_Duration=7, Process_End_Time=1463531932570, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=18]]
2016-05-18 00:38:52,574 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531932233_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:52,574 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, taskAttemptId=attempt_1463531932233_0001_1_01_000000_0, creationTime=1463531932507, allocationTime=1463531932508, startTime=1463531932510, finishTime=1463531932574, timeTaken=64, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:52,574 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:52,574 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:52,574 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531932233_0001_1_01_000000_0
2016-05-18 00:38:52,574 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,575 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932233_0001_00_000002
2016-05-18 00:38:52,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:TASK_FINISHED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, taskId=task_1463531932233_0001_1_01_000000, startTime=1463531932510, finishTime=1463531932574, timeTaken=64, status=SUCCEEDED, successfulAttemptID=attempt_1463531932233_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531932563, Process_Duration=7, Process_End_Time=1463531932570, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18
2016-05-18 00:38:52,575 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:52,576 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C]
2016-05-18 00:38:52,577 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:VERTEX_FINISHED]: vertexName=A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C, vertexId=vertex_1463531932233_0001_1_01, initRequestedTime=1463531932405, initedTime=1463531932411, startRequestedTime=1463531932422, startedTime=1463531932422, finishTime=1463531932576, timeTaken=154, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531932563, Process_Duration=7, Process_End_Time=1463531932570, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18, vertexStats=firstTaskStartTime=1463531932510, firstTasksToStart=[ task_1463531932233_0001_1_01_000000 ], lastTaskFinishTime=1463531932574, lastTasksToFinish=[ task_1463531932233_0001_1_01_000000 ], minTaskDuration=64, maxTaskDuration=64, avgTaskDuration=64.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531932233_0001_1_01_000000 ], longestDurationTasks=[ task_1463531932233_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:52,577 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:52,578 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531932233_0001_00_000002, stoppedTime=1463531932578, exitStatus=0
2016-05-18 00:38:52,578 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:52,578 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:52,578 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531932233_0001_1_00 [B389936F7B9544EFBE78775452C4B1B7]
2016-05-18 00:38:52,579 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531932233_0001_1
2016-05-18 00:38:52,579 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531932233_0001_1
2016-05-18 00:38:52,579 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932233_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:52,579 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 7F1E6629832F471589AF97168BEDF874 for vertex: vertex_1463531932233_0001_1_01 [A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C], outputName: 7F1E6629832F471589AF97168BEDF874
2016-05-18 00:38:52,579 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531932233_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:52,585 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531932233_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:52,585 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932233_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:52,585 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531932233_0001_00_000002, Exiting
2016-05-18 00:38:52,585 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531932233_0001_00_000002
2016-05-18 00:38:52,586 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531932233_0001_00_000002
2016-05-18 00:38:52,586 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531932233_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:52,586 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531932233_0001_00_000002 completed successfully
2016-05-18 00:38:52,587 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:7F1E6629832F471589AF97168BEDF874 of vertex/vertexGroup:A9A0C63D5FD64F2A8DC3F4BF7B9BFE0C isVertexGroupOutput:false
2016-05-18 00:38:52,587 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:52,590 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531932233_0001_1, queueSize=0
2016-05-18 00:38:52,590 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932233_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531932233_0001_1, startTime=1463531932402, finishTime=1463531932587, timeTaken=185, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=350, AM_GC_TIME_MILLIS=12, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=31, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063865050, Process_Duration=9, Process_End_Time=2927063865059, Read_Duration=2, Tuples_Read=26, Tuples_Written=31, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=18
2016-05-18 00:38:52,590 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531932233_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:52,591 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932233_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:52 Completed Dag: dag_1463531932233_0001_1
2016-05-18 00:38:52,591 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531932233_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:52,591 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531932233_0001_1_post
2016-05-18 00:38:52,591 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:52,591 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:52,592 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:52,592 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2, with id=dag_1463531932233_0001_1
2016-05-18 00:38:52,592 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB] test/(1/1) ...testSimpleBuffer2/simple2, with id=dag_1463531932233_0001_1
2016-05-18 00:38:52,600 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463531932233_0001
2016-05-18 00:38:52,602 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:52,602 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:52,602 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:52,602 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:52,602 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:52,603 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:52,603 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:52,604 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:52,604 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:52,604 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:52,604 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:52,604 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:52,605 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:52,607 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:52,610 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0E348F8EFB2469BBF20FA8FB140ABD9/D075017FF5CA45708137FFAB6CBAB9EB/.tez/application_1463531932233_0001
2016-05-18 00:38:52,611 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2/_temporary
2016-05-18 00:38:52,611 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.398
2016-05-18 00:38:52,689 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-18 00:38:52,722 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:52,723 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:52,774 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.023
2016-05-18 00:38:52,794 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.042
2016-05-18 00:38:52,795 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:52,796 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:52,796 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:52,822 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt
2016-05-18 00:38:52,834 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-18 00:38:52,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:52,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 00:38:52,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-18 00:38:52,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:52,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:52,897 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:52,899 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer3/simple3
2016-05-18 00:38:52,899 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:52,914 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:52,918 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:52,919 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:52,919 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:52,929 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001 doesn't exist and is created
2016-05-18 00:38:52,943 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:52,944 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:52,945 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd
2016-05-18 00:38:52,955 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531932919_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:52,955 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:52,955 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:52,956 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:52,956 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:52,956 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:52,957 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:52,957 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:52,959 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:52,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:52,960 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:52,960 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:52,961 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:52,962 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:52,962 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:52,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-log-dir
2016-05-18 00:38:52,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-log-dir/history.txt.appattempt_1463531932919_0001_000000, maxErrors=10
2016-05-18 00:38:52,962 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:52,963 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531932919_0001_000000, appSubmitTime=1463531932954, launchTime=1463531932954
2016-05-18 00:38:52,967 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:52,970 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:59436
2016-05-18 00:38:52,970 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531932919
2016-05-18 00:38:52,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531932919_0001_000000, startTime=1463531932974
2016-05-18 00:38:52,974 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:53,044 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:53,044 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:53,044 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463531932919_0001, dagName=[DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 00:38:53,064 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 00:38:53,064 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:53,065 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531932919_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-log-dir/dag_1463531932919_0001_1.dot
2016-05-18 00:38:53,066 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-log-dir/dag_1463531932919_0001_1-tez-dag.pb.txt
2016-05-18 00:38:53,073 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531932919_0001_1
2016-05-18 00:38:53,073 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,073 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 00:38:53 Running Dag: dag_1463531932919_0001_1
2016-05-18 00:38:53,083 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531932919_0001_1, submitTime=1463531933064
2016-05-18 00:38:53,084 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001/tez-conf.pb" } size: 36903 timestamp: 1463531932000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:53,085 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,085 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:53,086 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001/tez-conf.pb" } size: 36903 timestamp: 1463531932000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:53,087 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,087 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:53,088 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:53,089 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531932919_0001_1, initTime=1463531933088
2016-05-18 00:38:53,089 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932919_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:53,090 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:53,090 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:DAG_STARTED]: dagID=dag_1463531932919_0001_1, startTime=1463531933090
2016-05-18 00:38:53,090 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932919_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:53,090 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DC27D4563B51432288CC40C90361F663 : {A9FD632999CD4B8DB88683D6CAC97C47={InputName=A9FD632999CD4B8DB88683D6CAC97C47}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:53,090 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A9FD632999CD4B8DB88683D6CAC97C47, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:53,091 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,091 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,091 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,091 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,092 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:53,092 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A9FD632999CD4B8DB88683D6CAC97C47 on vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,092 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463531932919_0001, dagName=[DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 00:38:53,092 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,094 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A9FD632999CD4B8DB88683D6CAC97C47 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:53,097 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:53,097 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 30 Min splitLength: 52428800 New desired splits: 1 Total length: 30 Original splits: 2
2016-05-18 00:38:53,097 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 30 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 30 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:53,097 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:53,098 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:53,098 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,098 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A9FD632999CD4B8DB88683D6CAC97C47 on vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,099 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,100 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531932919_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:53,100 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0], numAdditionalOutputs=1
2016-05-18 00:38:53,101 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=7C61CFB0A42A420DAB0B64DAF0FBA9DD, vertex=vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:53,102 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D42696253187481FBFBC3D110D25EAE0:7C61CFB0A42A420DAB0B64DAF0FBA9DD using oldmapred API
2016-05-18 00:38:53,103 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=7C61CFB0A42A420DAB0B64DAF0FBA9DD, vertexName=D42696253187481FBFBC3D110D25EAE0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:53,106 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D42696253187481FBFBC3D110D25EAE0, vertexId=vertex_1463531932919_0001_1_01, initRequestedTime=1463531933092, initedTime=1463531933100, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:53,106 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:53,108 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:53,108 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A9FD632999CD4B8DB88683D6CAC97C47=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:53,108 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] parallelism set to 1
2016-05-18 00:38:53,109 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:53,109 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DC27D4563B51432288CC40C90361F663, vertexId=vertex_1463531932919_0001_1_00, initRequestedTime=1463531933090, initedTime=1463531933109, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:53,109 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] with distanceFromRoot: 0
2016-05-18 00:38:53,109 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:53,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531932919_0001_1_00, startRequestedTime=1463531933092, startedTime=1463531933110
2016-05-18 00:38:53,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531932919_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:53,110 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:53,110 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531932919_0001_1_00 for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] numStartedSources: 1 numSources: 1
2016-05-18 00:38:53,110 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] with distanceFromRoot: 1
2016-05-18 00:38:53,111 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531932919_0001_1_01, startRequestedTime=1463531933110, startedTime=1463531933110
2016-05-18 00:38:53,111 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:53,116 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DC27D4563B51432288CC40C90361F663
2016-05-18 00:38:53,116 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D42696253187481FBFBC3D110D25EAE0 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:53,116 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,118 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DC27D4563B51432288CC40C90361F663 in vertex: D42696253187481FBFBC3D110D25EAE0 numBipartiteSourceTasks: 1
2016-05-18 00:38:53,118 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_STARTED]: vertexName=DC27D4563B51432288CC40C90361F663, taskId=task_1463531932919_0001_1_00_000000, scheduledTime=1463531933118, launchTime=1463531933118
2016-05-18 00:38:53,118 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,119 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:53,121 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531932919_0001_00_000001, launchTime=1463531933121
2016-05-18 00:38:53,121 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531932919_0001, with dag id: dag_1463531932919_0001_1
2016-05-18 00:38:53,123 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932919_0001_00_000001
2016-05-18 00:38:53,125 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932919_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,125 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531932919_0001_1_00_000000_0] started. Is using containerId: [container_1463531932919_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:53,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DC27D4563B51432288CC40C90361F663, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0, startTime=1463531933125, containerId=container_1463531932919_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:53 Starting to run new task attempt: attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,130 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,130 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,130 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:53,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: DC27D4563B51432288CC40C90361F663, VertexParallelism: 1, TaskAttemptID:attempt_1463531932919_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A9FD632999CD4B8DB88683D6CAC97C47, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D42696253187481FBFBC3D110D25EAE0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:53,131 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:53,132 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,135 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531932919_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:53,135 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531932919_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:53,137 INFO input.MRInput (MRInput.java:initialize(440)) - A9FD632999CD4B8DB88683D6CAC97C47 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:53,137 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A9FD632999CD4B8DB88683D6CAC97C47
2016-05-18 00:38:53,138 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:53,145 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DC27D4563B51432288CC40C90361F663, ordinal: 0
2016-05-18 00:38:53,145 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DC27D4563B51432288CC40C90361F663, mem on start (mb), free: 145, total: 489, max: 672
2016-05-18 00:38:53,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:53,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:53,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:53,146 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:53,146 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A9FD632999CD4B8DB88683D6CAC97C47:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D42696253187481FBFBC3D110D25EAE0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:53,146 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:53,147 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:53,147 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:53,147 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A9FD632999CD4B8DB88683D6CAC97C47:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D42696253187481FBFBC3D110D25EAE0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:53,147 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:53,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A9FD632999CD4B8DB88683D6CAC97C47 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:53,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:53,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:53,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A9FD632999CD4B8DB88683D6CAC97C47
2016-05-18 00:38:53,149 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:53,149 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,152 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 00:38:53,153 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"] streamed: true, id: A9FD632999CD4B8DB88683D6CAC97C47
2016-05-18 00:38:53,153 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: F2003E56D55B4BCFAD19F19ADA6638D6
2016-05-18 00:38:53,153 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] F2003E56D55B4BCFAD19F19ADA6638D6
2016-05-18 00:38:53,154 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt:0+15, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt:15+15], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=30}
2016-05-18 00:38:53,155 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A9FD632999CD4B8DB88683D6CAC97C47 initialized RecordReader from event
2016-05-18 00:38:53,156 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D42696253187481FBFBC3D110D25EAE0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@224de8fc, comparator=cascading.tuple.hadoop.util.TupleComparator@28fb09fc, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:53,156 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D42696253187481FBFBC3D110D25EAE0: , UsingHashComparator=false
2016-05-18 00:38:53,169 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:53,169 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:53,169 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D42696253187481FBFBC3D110D25EAE0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:53,169 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D42696253187481FBFBC3D110D25EAE0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:53,170 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 00:38:53,171 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DC27D4563B51432288CC40C90361F663, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:53,174 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,174 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DC27D4563B51432288CC40C90361F663, mem on close (mb), free: 44, total: 489, max: 672
2016-05-18 00:38:53,175 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D42696253187481FBFBC3D110D25EAE0: Starting flush of map output
2016-05-18 00:38:53,175 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D42696253187481FBFBC3D110D25EAE0: Span0.length = 5, perItem = 21
2016-05-18 00:38:53,175 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D42696253187481FBFBC3D110D25EAE0: done sorting span=0, length=5, time=0
2016-05-18 00:38:53,176 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D42696253187481FBFBC3D110D25EAE0: Heap = SpanIterator<0:4> (span=Span[16777216,105]),
2016-05-18 00:38:53,177 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D42696253187481FBFBC3D110D25EAE0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-local-dir/output/attempt_1463531932919_0001_1_00_000000_0_10037_0/file.out
2016-05-18 00:38:53,180 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:53 Completed running task attempt: attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,181 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:53,181 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531932919_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:53,184 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531932919_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531933171, Process_Duration=3, Process_End_Time=1463531933174, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 00:38:53,185 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531932919_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,185 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,185 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:53,185 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,185 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932919_0001_00_000001
2016-05-18 00:38:53,186 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DC27D4563B51432288CC40C90361F663, taskAttemptId=attempt_1463531932919_0001_1_00_000000_0, creationTime=1463531933118, allocationTime=1463531933120, startTime=1463531933125, finishTime=1463531933185, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:53,187 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531932919_0001_1_00_000000_0
2016-05-18 00:38:53,188 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_FINISHED]: vertexName=DC27D4563B51432288CC40C90361F663, taskId=task_1463531932919_0001_1_00_000000, startTime=1463531933125, finishTime=1463531933187, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1463531932919_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531933171, Process_Duration=3, Process_End_Time=1463531933174, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 00:38:53,188 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:53,188 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,191 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_FINISHED]: vertexName=DC27D4563B51432288CC40C90361F663, vertexId=vertex_1463531932919_0001_1_00, initRequestedTime=1463531933090, initedTime=1463531933109, startRequestedTime=1463531933092, startedTime=1463531933110, finishTime=1463531933188, timeTaken=78, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531933171, Process_Duration=3, Process_End_Time=1463531933174, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463531933125, firstTasksToStart=[ task_1463531932919_0001_1_00_000000 ], lastTaskFinishTime=1463531933185, lastTasksToFinish=[ task_1463531932919_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531932919_0001_1_00_000000 ], longestDurationTasks=[ task_1463531932919_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:53,191 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:53,191 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531932919_0001_00_000001, stoppedTime=1463531933191, exitStatus=0
2016-05-18 00:38:53,191 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] attempt: attempt_1463531932919_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:53,192 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:53,192 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:53,192 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D42696253187481FBFBC3D110D25EAE0
2016-05-18 00:38:53,192 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531932919_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:53,192 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DC27D4563B51432288CC40C90361F663, destinationVertexName=D42696253187481FBFBC3D110D25EAE0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:53,192 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,194 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_STARTED]: vertexName=D42696253187481FBFBC3D110D25EAE0, taskId=task_1463531932919_0001_1_01_000000, scheduledTime=1463531933194, launchTime=1463531933194
2016-05-18 00:38:53,194 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531932919_0001_00_000002, launchTime=1463531933196
2016-05-18 00:38:53,197 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932919_0001_00_000002
2016-05-18 00:38:53,199 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531932919_0001_1_01_000000_0] started. Is using containerId: [container_1463531932919_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:53,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D42696253187481FBFBC3D110D25EAE0, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0, startTime=1463531933199, containerId=container_1463531932919_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:53,199 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932919_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53 Starting to run new task attempt: attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,200 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,200 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,200 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:53,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: D42696253187481FBFBC3D110D25EAE0, VertexParallelism: 1, TaskAttemptID:attempt_1463531932919_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DC27D4563B51432288CC40C90361F663, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=7C61CFB0A42A420DAB0B64DAF0FBA9DD, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:53,201 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:53,201 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531932919_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:53,201 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,201 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932919_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:53,202 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531932919_0001_00_000001, Exiting
2016-05-18 00:38:53,202 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531932919_0001_00_000001
2016-05-18 00:38:53,202 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531932919_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,202 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531932919_0001_00_000001
2016-05-18 00:38:53,203 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531932919_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:53,204 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:53,205 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531932919_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:53,214 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531932919_0001_00_000001 completed successfully
2016-05-18 00:38:53,216 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:53,229 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D42696253187481FBFBC3D110D25EAE0, ordinal: 1
2016-05-18 00:38:53,229 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D42696253187481FBFBC3D110D25EAE0, mem on start (mb), free: 253, total: 490, max: 672
2016-05-18 00:38:53,229 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:53,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:53,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:53,230 INFO output.MROutput (MROutput.java:initialize(421)) - 7C61CFB0A42A420DAB0B64DAF0FBA9DD: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:53,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:53,230 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DC27D4563B51432288CC40C90361F663:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [7C61CFB0A42A420DAB0B64DAF0FBA9DD:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:38:53,231 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:53,231 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:53,231 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:53,231 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DC27D4563B51432288CC40C90361F663:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [7C61CFB0A42A420DAB0B64DAF0FBA9DD:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:38:53,231 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:53,231 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DC27D4563B51432288CC40C90361F663 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:53,232 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DC27D4563B51432288CC40C90361F663: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:53,232 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DC27D4563B51432288CC40C90361F663: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:53,233 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DC27D4563B51432288CC40C90361F663 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:53,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:53,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DC27D4563B51432288CC40C90361F663
2016-05-18 00:38:53,235 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:53,235 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:53,235 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,237 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DC27D4563B51432288CC40C90361F663: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:53,238 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:53,239 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: F2003E56D55B4BCFAD19F19ADA6638D6
2016-05-18 00:38:53,240 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531932919_0001_1_00_000000_0_10037} to DISK_DIRECT, csize=125, dsize=121, EndTime=1463531933240, TimeTaken=1, Rate=0.12 MB/s
2016-05-18 00:38:53,240 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DC27D4563B51432288CC40C90361F663
2016-05-18 00:38:53,240 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:53,241 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DC27D4563B51432288CC40C90361F663, wasInterrupted=false
2016-05-18 00:38:53,242 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:53,242 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:53,243 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001_wd/localmode-local-dir/output/attempt_1463531932919_0001_1_00_000000_0_10037/file.out, len=125, isLocal=true
2016-05-18 00:38:53,243 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 125 bytes from disk
2016-05-18 00:38:53,243 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:53,243 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:53,244 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2016-05-18 00:38:53,244 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DC27D4563B51432288CC40C90361F663
2016-05-18 00:38:53,244 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DC27D4563B51432288CC40C90361F663: Shuffle Runner thread complete
2016-05-18 00:38:53,245 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"], id: 7C61CFB0A42A420DAB0B64DAF0FBA9DD
2016-05-18 00:38:53,246 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-18 00:38:53,247 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] F2003E56D55B4BCFAD19F19ADA6638D6, for 1 inputs
2016-05-18 00:38:53,247 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D42696253187481FBFBC3D110D25EAE0, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:53,248 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DC27D4563B51432288CC40C90361F663: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:53,251 INFO output.MROutput (MROutput.java:close(526)) - 7C61CFB0A42A420DAB0B64DAF0FBA9DD closed
2016-05-18 00:38:53,251 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531932919_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:53,252 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,252 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D42696253187481FBFBC3D110D25EAE0, mem on close (mb), free: 251, total: 490, max: 672
2016-05-18 00:38:53,252 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DC27D4563B51432288CC40C90361F663
2016-05-18 00:38:53,253 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DC27D4563B51432288CC40C90361F663: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:53,254 INFO output.MROutput (MROutput.java:close(526)) - 7C61CFB0A42A420DAB0B64DAF0FBA9DD closed
2016-05-18 00:38:53,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:53 Completed running task attempt: attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,255 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:53,256 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531932919_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:53,256 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531932919_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531933247, Process_Duration=5, Process_End_Time=1463531933252, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 00:38:53,258 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531932919_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,258 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,258 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:53,258 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D42696253187481FBFBC3D110D25EAE0, taskAttemptId=attempt_1463531932919_0001_1_01_000000_0, creationTime=1463531933194, allocationTime=1463531933195, startTime=1463531933199, finishTime=1463531933259, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:53,259 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531932919_0001_1_01_000000_0
2016-05-18 00:38:53,260 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:TASK_FINISHED]: vertexName=D42696253187481FBFBC3D110D25EAE0, taskId=task_1463531932919_0001_1_01_000000, startTime=1463531933199, finishTime=1463531933259, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463531932919_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531933247, Process_Duration=5, Process_End_Time=1463531933252, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 00:38:53,260 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:53,260 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0]
2016-05-18 00:38:53,260 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531932919_0001_00_000002
2016-05-18 00:38:53,261 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:VERTEX_FINISHED]: vertexName=D42696253187481FBFBC3D110D25EAE0, vertexId=vertex_1463531932919_0001_1_01, initRequestedTime=1463531933092, initedTime=1463531933100, startRequestedTime=1463531933110, startedTime=1463531933110, finishTime=1463531933260, timeTaken=150, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531933247, Process_Duration=5, Process_End_Time=1463531933252, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463531933199, firstTasksToStart=[ task_1463531932919_0001_1_01_000000 ], lastTaskFinishTime=1463531933259, lastTasksToFinish=[ task_1463531932919_0001_1_01_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531932919_0001_1_01_000000 ], longestDurationTasks=[ task_1463531932919_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:53,262 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:53,262 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531932919_0001_00_000002, stoppedTime=1463531933262, exitStatus=0
2016-05-18 00:38:53,262 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:53,262 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:53,262 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531932919_0001_1_00 [DC27D4563B51432288CC40C90361F663]
2016-05-18 00:38:53,262 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531932919_0001_1
2016-05-18 00:38:53,263 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531932919_0001_1
2016-05-18 00:38:53,263 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 7C61CFB0A42A420DAB0B64DAF0FBA9DD for vertex: vertex_1463531932919_0001_1_01 [D42696253187481FBFBC3D110D25EAE0], outputName: 7C61CFB0A42A420DAB0B64DAF0FBA9DD
2016-05-18 00:38:53,263 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932919_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:53,264 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531932919_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:53,269 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:7C61CFB0A42A420DAB0B64DAF0FBA9DD of vertex/vertexGroup:D42696253187481FBFBC3D110D25EAE0 isVertexGroupOutput:false
2016-05-18 00:38:53,269 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:53,270 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531932919_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:53,271 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531932919_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:53,271 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531932919_0001_00_000002, Exiting
2016-05-18 00:38:53,271 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531932919_0001_00_000002
2016-05-18 00:38:53,271 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531932919_0001_00_000002
2016-05-18 00:38:53,271 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531932919_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:53,271 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531932919_0001_00_000002 completed successfully
2016-05-18 00:38:53,273 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531932919_0001_1, queueSize=0
2016-05-18 00:38:53,274 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531932919_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531932919_0001_1, startTime=1463531933090, finishTime=1463531933269, timeTaken=179, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=340, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=10, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063866418, Process_Duration=8, Process_End_Time=2927063866426, Read_Duration=1, Tuples_Read=10, Tuples_Written=10, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, Tuples_Written=5
2016-05-18 00:38:53,275 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531932919_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:53,275 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531932919_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:53 Completed Dag: dag_1463531932919_0001_1
2016-05-18 00:38:53,275 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531932919_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:53,276 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531932919_0001_1_post
2016-05-18 00:38:53,276 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,276 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:53,277 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:53,277 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3, with id=dag_1463531932919_0001_1
2016-05-18 00:38:53,277 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2] test/(1/1) ...testSimpleBuffer3/simple3, with id=dag_1463531932919_0001_1
2016-05-18 00:38:53,283 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463531932919_0001
2016-05-18 00:38:53,285 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:53,285 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:53,285 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:53,285 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:53,285 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:53,286 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:53,286 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:53,287 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:53,287 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:53,287 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:53,287 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:53,288 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:53,288 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:53,288 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:53,292 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC5FA0AC83B94FB782CA26C45F7C5A2F/FAC935EA4C094F5C9A7EC866E5B42FD2/.tez/application_1463531932919_0001
2016-05-18 00:38:53,292 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3/_temporary
2016-05-18 00:38:53,293 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.396
2016-05-18 00:38:53,373 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-18 00:38:53,402 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:53,403 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:53,455 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 00:38:53,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.029
2016-05-18 00:38:53,465 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:53,465 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:53,465 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:53,482 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:53,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-18 00:38:53,549 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:53,550 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:53,550 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-18 00:38:53,550 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:53,551 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:53,551 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:53,551 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferReverse/firstn
2016-05-18 00:38:53,552 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:53,565 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:53,569 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:53,570 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:53,570 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:53,578 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001 doesn't exist and is created
2016-05-18 00:38:53,593 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:53,593 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:53,594 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd
2016-05-18 00:38:53,604 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531933569_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:53,605 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:53,605 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:53,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:53,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:53,606 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:53,607 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:53,608 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:53,609 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:53,609 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:53,610 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:53,610 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:53,610 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:53,610 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:53,610 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:53,611 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:53,611 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:53,611 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:53,612 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:53,612 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:53,613 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:53,614 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:53,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:53,616 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-log-dir
2016-05-18 00:38:53,616 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-log-dir/history.txt.appattempt_1463531933569_0001_000000, maxErrors=10
2016-05-18 00:38:53,616 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:53,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531933569_0001_000000, appSubmitTime=1463531933604, launchTime=1463531933604
2016-05-18 00:38:53,621 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:53,626 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56877
2016-05-18 00:38:53,627 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531933569
2016-05-18 00:38:53,629 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531933569_0001_000000, startTime=1463531933628
2016-05-18 00:38:53,629 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:53,694 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:53,694 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:53,694 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463531933569_0001, dagName=[4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 00:38:53,712 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 00:38:53,712 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:53,713 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531933569_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-log-dir/dag_1463531933569_0001_1.dot
2016-05-18 00:38:53,714 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-log-dir/dag_1463531933569_0001_1-tez-dag.pb.txt
2016-05-18 00:38:53,720 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531933569_0001_1
2016-05-18 00:38:53,720 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,720 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 00:38:53 Running Dag: dag_1463531933569_0001_1
2016-05-18 00:38:53,729 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531933569_0001_1, submitTime=1463531933712
2016-05-18 00:38:53,731 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001/tez-conf.pb" } size: 36903 timestamp: 1463531933000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:53,731 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,731 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:53,732 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001/tez-conf.pb" } size: 36903 timestamp: 1463531933000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:53,732 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,732 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:53,732 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:53,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531933569_0001_1, initTime=1463531933733
2016-05-18 00:38:53,733 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531933569_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:53,734 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:53,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:DAG_STARTED]: dagID=dag_1463531933569_0001_1, startTime=1463531933734
2016-05-18 00:38:53,734 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531933569_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:53,734 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CAC4B61AEE1344D391EC197EFBB53ACC : {F518B5E6B27B4935AD35C910B99C7B91={InputName=F518B5E6B27B4935AD35C910B99C7B91}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:53,734 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: F518B5E6B27B4935AD35C910B99C7B91, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:53,734 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463531933569_0001, dagName=[4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 00:38:53,734 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,735 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,735 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,735 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,736 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:53,736 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,740 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:53,740 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,741 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,741 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: F518B5E6B27B4935AD35C910B99C7B91 on vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,743 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531933569_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:53,743 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05], numAdditionalOutputs=1
2016-05-18 00:38:53,743 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=D6D519221B184D6F9627E96C525CAA44, vertex=vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:53,745 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input F518B5E6B27B4935AD35C910B99C7B91 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:53,746 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F3AAD393E1C846B4B4521F0690393A05:D6D519221B184D6F9627E96C525CAA44 using oldmapred API
2016-05-18 00:38:53,746 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=D6D519221B184D6F9627E96C525CAA44, vertexName=F3AAD393E1C846B4B4521F0690393A05, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:53,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, vertexId=vertex_1463531933569_0001_1_01, initRequestedTime=1463531933736, initedTime=1463531933743, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:53,747 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:53,748 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:53,748 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:53,748 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:53,748 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:53,749 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: F518B5E6B27B4935AD35C910B99C7B91 on vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,751 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:53,751 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {F518B5E6B27B4935AD35C910B99C7B91=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:53,751 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] parallelism set to 1
2016-05-18 00:38:53,751 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:53,752 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, vertexId=vertex_1463531933569_0001_1_00, initRequestedTime=1463531933734, initedTime=1463531933752, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:53,752 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] with distanceFromRoot: 0
2016-05-18 00:38:53,752 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:53,753 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531933569_0001_1_00, startRequestedTime=1463531933736, startedTime=1463531933753
2016-05-18 00:38:53,753 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531933569_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:53,754 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:53,754 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531933569_0001_1_00 for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] numStartedSources: 1 numSources: 1
2016-05-18 00:38:53,755 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] with distanceFromRoot: 1
2016-05-18 00:38:53,754 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CAC4B61AEE1344D391EC197EFBB53ACC
2016-05-18 00:38:53,755 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,756 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531933569_0001_1_01, startRequestedTime=1463531933754, startedTime=1463531933755
2016-05-18 00:38:53,756 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531933569_0001, with dag id: dag_1463531933569_0001_1
2016-05-18 00:38:53,756 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:53,758 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F3AAD393E1C846B4B4521F0690393A05 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:53,758 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_STARTED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, taskId=task_1463531933569_0001_1_00_000000, scheduledTime=1463531933758, launchTime=1463531933758
2016-05-18 00:38:53,760 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,760 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CAC4B61AEE1344D391EC197EFBB53ACC in vertex: F3AAD393E1C846B4B4521F0690393A05 numBipartiteSourceTasks: 1
2016-05-18 00:38:53,762 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:53,764 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531933569_0001_00_000001, launchTime=1463531933764
2016-05-18 00:38:53,765 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531933569_0001_00_000001
2016-05-18 00:38:53,766 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531933569_0001_1_00_000000_0] started. Is using containerId: [container_1463531933569_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:53,767 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0, startTime=1463531933766, containerId=container_1463531933569_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:53,767 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531933569_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53 Starting to run new task attempt: attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,768 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,768 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,768 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:53,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn, VertexName: CAC4B61AEE1344D391EC197EFBB53ACC, VertexParallelism: 1, TaskAttemptID:attempt_1463531933569_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F518B5E6B27B4935AD35C910B99C7B91, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F3AAD393E1C846B4B4521F0690393A05, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:53,769 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:53,769 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,772 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531933569_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:53,772 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531933569_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:53,776 INFO input.MRInput (MRInput.java:initialize(440)) - F518B5E6B27B4935AD35C910B99C7B91 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:53,776 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:53,776 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: F518B5E6B27B4935AD35C910B99C7B91
2016-05-18 00:38:53,782 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CAC4B61AEE1344D391EC197EFBB53ACC, ordinal: 0
2016-05-18 00:38:53,782 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CAC4B61AEE1344D391EC197EFBB53ACC, mem on start (mb), free: 147, total: 490, max: 672
2016-05-18 00:38:53,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:53,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:53,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:53,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:53,783 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F518B5E6B27B4935AD35C910B99C7B91:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F3AAD393E1C846B4B4521F0690393A05:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:53,783 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:53,784 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:53,784 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:53,784 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F518B5E6B27B4935AD35C910B99C7B91:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F3AAD393E1C846B4B4521F0690393A05:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:53,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:53,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F518B5E6B27B4935AD35C910B99C7B91
2016-05-18 00:38:53,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F518B5E6B27B4935AD35C910B99C7B91 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:53,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:53,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:53,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:53,785 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,788 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:53,788 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: F518B5E6B27B4935AD35C910B99C7B91
2016-05-18 00:38:53,789 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: DB85C92B5D6946DFBEDAFC962BCF58C1
2016-05-18 00:38:53,789 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] DB85C92B5D6946DFBEDAFC962BCF58C1
2016-05-18 00:38:53,792 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F3AAD393E1C846B4B4521F0690393A05 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@188f5e03, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@6c65238d, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:53,792 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F3AAD393E1C846B4B4521F0690393A05: , UsingHashComparator=false
2016-05-18 00:38:53,794 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:53,795 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - F518B5E6B27B4935AD35C910B99C7B91 initialized RecordReader from event
2016-05-18 00:38:53,805 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:53,806 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:53,807 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F3AAD393E1C846B4B4521F0690393A05: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:53,807 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F3AAD393E1C846B4B4521F0690393A05: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:53,808 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:53,808 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CAC4B61AEE1344D391EC197EFBB53ACC, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:53,812 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,812 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CAC4B61AEE1344D391EC197EFBB53ACC, mem on close (mb), free: 45, total: 490, max: 672
2016-05-18 00:38:53,813 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F3AAD393E1C846B4B4521F0690393A05: Starting flush of map output
2016-05-18 00:38:53,813 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F3AAD393E1C846B4B4521F0690393A05: Span0.length = 13, perItem = 22
2016-05-18 00:38:53,814 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F3AAD393E1C846B4B4521F0690393A05: done sorting span=0, length=13, time=1
2016-05-18 00:38:53,814 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F3AAD393E1C846B4B4521F0690393A05: Heap = SpanIterator<0:12> (span=Span[16777216,286]),
2016-05-18 00:38:53,816 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F3AAD393E1C846B4B4521F0690393A05: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-local-dir/output/attempt_1463531933569_0001_1_00_000000_0_10043_0/file.out
2016-05-18 00:38:53,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:53 Completed running task attempt: attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,820 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:53,820 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531933569_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:53,821 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531933569_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531933808, Process_Duration=4, Process_End_Time=1463531933812, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:53,822 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531933569_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,822 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:53,823 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, taskAttemptId=attempt_1463531933569_0001_1_00_000000_0, creationTime=1463531933760, allocationTime=1463531933763, startTime=1463531933766, finishTime=1463531933822, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:53,823 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:53,823 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,823 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531933569_0001_1_00_000000_0
2016-05-18 00:38:53,824 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531933569_0001_00_000001
2016-05-18 00:38:53,825 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_FINISHED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, taskId=task_1463531933569_0001_1_00_000000, startTime=1463531933766, finishTime=1463531933823, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463531933569_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531933808, Process_Duration=4, Process_End_Time=1463531933812, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:53,825 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:53,825 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:53,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_FINISHED]: vertexName=CAC4B61AEE1344D391EC197EFBB53ACC, vertexId=vertex_1463531933569_0001_1_00, initRequestedTime=1463531933734, initedTime=1463531933752, startRequestedTime=1463531933736, startedTime=1463531933753, finishTime=1463531933825, timeTaken=72, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531933808, Process_Duration=4, Process_End_Time=1463531933812, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531933766, firstTasksToStart=[ task_1463531933569_0001_1_00_000000 ], lastTaskFinishTime=1463531933822, lastTasksToFinish=[ task_1463531933569_0001_1_00_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531933569_0001_1_00_000000 ], longestDurationTasks=[ task_1463531933569_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:53,827 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:53,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531933569_0001_00_000001, stoppedTime=1463531933827, exitStatus=0
2016-05-18 00:38:53,827 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] attempt: attempt_1463531933569_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:53,828 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:53,828 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F3AAD393E1C846B4B4521F0690393A05
2016-05-18 00:38:53,828 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:53,828 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CAC4B61AEE1344D391EC197EFBB53ACC, destinationVertexName=F3AAD393E1C846B4B4521F0690393A05. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:53,828 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531933569_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:53,828 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,829 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_STARTED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, taskId=task_1463531933569_0001_1_01_000000, scheduledTime=1463531933829, launchTime=1463531933829
2016-05-18 00:38:53,829 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,833 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531933569_0001_00_000002, launchTime=1463531933833
2016-05-18 00:38:53,834 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531933569_0001_00_000002
2016-05-18 00:38:53,835 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531933569_0001_1_01_000000_0] started. Is using containerId: [container_1463531933569_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:53,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0, startTime=1463531933834, containerId=container_1463531933569_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:53,835 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531933569_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:53,835 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531933569_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:53,835 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531933569_0001_00_000001, Exiting
2016-05-18 00:38:53,835 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531933569_0001_00_000001
2016-05-18 00:38:53,835 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531933569_0001_00_000001
2016-05-18 00:38:53,836 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531933569_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:53,836 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531933569_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:53,837 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531933569_0001_00_000001 completed successfully
2016-05-18 00:38:53 Starting to run new task attempt: attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:53,838 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:53,838 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:53,838 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:53,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn, VertexName: F3AAD393E1C846B4B4521F0690393A05, VertexParallelism: 1, TaskAttemptID:attempt_1463531933569_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CAC4B61AEE1344D391EC197EFBB53ACC, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D6D519221B184D6F9627E96C525CAA44, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:53,839 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:53,840 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:53,840 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531933569_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:53,841 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531933569_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:53,843 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:53,845 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:53,850 INFO output.MROutput (MROutput.java:initialize(421)) - D6D519221B184D6F9627E96C525CAA44: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:53,852 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F3AAD393E1C846B4B4521F0690393A05, ordinal: 1
2016-05-18 00:38:53,852 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3AAD393E1C846B4B4521F0690393A05, mem on start (mb), free: 40, total: 490, max: 672
2016-05-18 00:38:53,852 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:53,852 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:53,852 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:53,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:53,853 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D6D519221B184D6F9627E96C525CAA44:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [CAC4B61AEE1344D391EC197EFBB53ACC:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:38:53,853 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:53,853 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:53,853 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:53,853 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D6D519221B184D6F9627E96C525CAA44:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [CAC4B61AEE1344D391EC197EFBB53ACC:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:38:53,854 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:53,854 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CAC4B61AEE1344D391EC197EFBB53ACC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:53,854 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:53,855 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:53,855 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CAC4B61AEE1344D391EC197EFBB53ACC: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:53,855 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CAC4B61AEE1344D391EC197EFBB53ACC: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:53,856 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CAC4B61AEE1344D391EC197EFBB53ACC with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:53,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CAC4B61AEE1344D391EC197EFBB53ACC
2016-05-18 00:38:53,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:53,859 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:53,859 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CAC4B61AEE1344D391EC197EFBB53ACC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:54,092 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:38:54,092 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:38:54,095 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531933569_0001_1_00_000000_0_10043} to DISK_DIRECT, csize=322, dsize=318, EndTime=1463531934095, TimeTaken=234, Rate=0.00 MB/s
2016-05-18 00:38:54,095 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CAC4B61AEE1344D391EC197EFBB53ACC
2016-05-18 00:38:54,095 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:54,096 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CAC4B61AEE1344D391EC197EFBB53ACC, wasInterrupted=false
2016-05-18 00:38:54,097 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:54,097 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:54,097 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:54,098 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: DB85C92B5D6946DFBEDAFC962BCF58C1
2016-05-18 00:38:54,098 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001_wd/localmode-local-dir/output/attempt_1463531933569_0001_1_00_000000_0_10043/file.out, len=322, isLocal=true
2016-05-18 00:38:54,098 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"], id: D6D519221B184D6F9627E96C525CAA44
2016-05-18 00:38:54,098 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 322 bytes from disk
2016-05-18 00:38:54,099 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:54,099 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-18 00:38:54,099 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:54,099 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] DB85C92B5D6946DFBEDAFC962BCF58C1, for 1 inputs
2016-05-18 00:38:54,099 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-18 00:38:54,099 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CAC4B61AEE1344D391EC197EFBB53ACC
2016-05-18 00:38:54,100 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CAC4B61AEE1344D391EC197EFBB53ACC: Shuffle Runner thread complete
2016-05-18 00:38:54,100 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F3AAD393E1C846B4B4521F0690393A05, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:54,100 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CAC4B61AEE1344D391EC197EFBB53ACC: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:54,103 INFO output.MROutput (MROutput.java:close(526)) - D6D519221B184D6F9627E96C525CAA44 closed
2016-05-18 00:38:54,104 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531933569_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:54,105 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:54,105 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3AAD393E1C846B4B4521F0690393A05, mem on close (mb), free: 341, total: 368, max: 672
2016-05-18 00:38:54,105 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CAC4B61AEE1344D391EC197EFBB53ACC
2016-05-18 00:38:54,105 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CAC4B61AEE1344D391EC197EFBB53ACC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:54,105 INFO output.MROutput (MROutput.java:close(526)) - D6D519221B184D6F9627E96C525CAA44 closed
2016-05-18 00:38:54,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:54 Completed running task attempt: attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:54,106 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:54,107 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531933569_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:54,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531933569_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=241, MERGE_PHASE_TIME=244, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531934100, Process_Duration=5, Process_End_Time=1463531934105, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 00:38:54,108 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531933569_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,108 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,108 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, taskAttemptId=attempt_1463531933569_0001_1_01_000000_0, creationTime=1463531933829, allocationTime=1463531933832, startTime=1463531933834, finishTime=1463531934108, timeTaken=274, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:54,109 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531933569_0001_1_01_000000_0
2016-05-18 00:38:54,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:TASK_FINISHED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, taskId=task_1463531933569_0001_1_01_000000, startTime=1463531933834, finishTime=1463531934109, timeTaken=275, status=SUCCEEDED, successfulAttemptID=attempt_1463531933569_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=241, MERGE_PHASE_TIME=244, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531934100, Process_Duration=5, Process_End_Time=1463531934105, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 00:38:54,109 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:54,110 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,111 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531933569_0001_00_000002
2016-05-18 00:38:54,110 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:54,111 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05]
2016-05-18 00:38:54,112 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:VERTEX_FINISHED]: vertexName=F3AAD393E1C846B4B4521F0690393A05, vertexId=vertex_1463531933569_0001_1_01, initRequestedTime=1463531933736, initedTime=1463531933743, startRequestedTime=1463531933754, startedTime=1463531933755, finishTime=1463531934111, timeTaken=356, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=241, MERGE_PHASE_TIME=244, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531934100, Process_Duration=5, Process_End_Time=1463531934105, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463531933834, firstTasksToStart=[ task_1463531933569_0001_1_01_000000 ], lastTaskFinishTime=1463531934108, lastTasksToFinish=[ task_1463531933569_0001_1_01_000000 ], minTaskDuration=274, maxTaskDuration=274, avgTaskDuration=274.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531933569_0001_1_01_000000 ], longestDurationTasks=[ task_1463531933569_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:54,113 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:54,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531933569_0001_00_000002, stoppedTime=1463531934113, exitStatus=0
2016-05-18 00:38:54,113 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:54,113 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:54,113 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531933569_0001_1_00 [CAC4B61AEE1344D391EC197EFBB53ACC]
2016-05-18 00:38:54,114 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531933569_0001_1
2016-05-18 00:38:54,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531933569_0001_1
2016-05-18 00:38:54,114 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531933569_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:54,114 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: D6D519221B184D6F9627E96C525CAA44 for vertex: vertex_1463531933569_0001_1_01 [F3AAD393E1C846B4B4521F0690393A05], outputName: D6D519221B184D6F9627E96C525CAA44
2016-05-18 00:38:54,114 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531933569_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:54,121 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:D6D519221B184D6F9627E96C525CAA44 of vertex/vertexGroup:F3AAD393E1C846B4B4521F0690393A05 isVertexGroupOutput:false
2016-05-18 00:38:54,121 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:54,121 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531933569_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:54,121 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531933569_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:54,122 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531933569_0001_00_000002, Exiting
2016-05-18 00:38:54,122 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531933569_0001_00_000002
2016-05-18 00:38:54,122 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531933569_0001_00_000002
2016-05-18 00:38:54,122 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531933569_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:54,122 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531933569_0001_00_000002 completed successfully
2016-05-18 00:38:54,124 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531933569_0001_1, queueSize=0
2016-05-18 00:38:54,125 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531933569_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531933569_0001_1, startTime=1463531933734, finishTime=1463531934121, timeTaken=387, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=720, AM_GC_TIME_MILLIS=229, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=241, MERGE_PHASE_TIME=244, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063867908, Process_Duration=9, Process_End_Time=2927063867917, Read_Duration=2, Tuples_Read=18, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 00:38:54,125 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531933569_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:54,126 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531933569_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:54 Completed Dag: dag_1463531933569_0001_1
2016-05-18 00:38:54,130 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531933569_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:54,131 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531933569_0001_1_post
2016-05-18 00:38:54,131 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,131 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:54,131 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:54,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn, with id=dag_1463531933569_0001_1
2016-05-18 00:38:54,132 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC] test/(1/1) ...irstNBufferReverse/firstn, with id=dag_1463531933569_0001_1
2016-05-18 00:38:54,134 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463531933569_0001
2016-05-18 00:38:54,138 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:54,138 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:54,138 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:54,138 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:54,138 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:54,139 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:54,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:54,139 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:54,140 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:54,140 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:54,141 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:54,141 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:54,141 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:54,143 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:54,147 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E211DF9E884485FA4FBF5E5EFF18461/E09711C04A394FD08D0D773F897D17AC/.tez/application_1463531933569_0001
2016-05-18 00:38:54,148 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn/_temporary
2016-05-18 00:38:54,148 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.598
2016-05-18 00:38:54,229 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-18 00:38:54,260 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:54,260 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:54,297 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-18 00:38:54,304 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.015
2016-05-18 00:38:54,304 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:38:54,305 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:38:54,335 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:38:54,336 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:38:54,381 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 00:38:54,406 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.040
2016-05-18 00:38:54,406 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:54,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:38:54,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:38:54,432 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 00:38:54,442 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn
2016-05-18 00:38:54,524 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 00:38:54,524 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:54,525 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-18 00:38:54,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 00:38:54,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 00:38:54,526 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 00:38:54,528 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testFirstNBuffer/firstn
2016-05-18 00:38:54,529 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:38:54,542 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 00:38:54,547 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:54,548 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:38:54,549 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:38:54,559 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001 doesn't exist and is created
2016-05-18 00:38:54,575 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:38:54,576 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:38:54,576 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd
2016-05-18 00:38:54,587 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463531934548_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:38:54,587 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:54,587 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:38:54,589 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:38:54,589 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:38:54,589 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:38:54,590 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:38:54,591 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:38:54,593 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:38:54,593 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:38:54,593 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:38:54,593 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:38:54,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:38:54,594 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:38:54,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:38:54,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:38:54,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:54,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:38:54,595 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:38:54,595 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:38:54,597 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:38:54,598 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:38:54,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:38:54,599 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-log-dir
2016-05-18 00:38:54,599 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-log-dir/history.txt.appattempt_1463531934548_0001_000000, maxErrors=10
2016-05-18 00:38:54,599 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:38:54,599 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463531934548_0001_000000, appSubmitTime=1463531934586, launchTime=1463531934586
2016-05-18 00:38:54,604 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:38:54,607 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:54366
2016-05-18 00:38:54,609 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463531934548
2016-05-18 00:38:54,611 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463531934548_0001_000000, startTime=1463531934611
2016-05-18 00:38:54,611 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:38:54,676 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:38:54,676 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:38:54,677 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463531934548_0001, dagName=[26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 00:38:54,695 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 00:38:54,695 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:38:54,696 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463531934548_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-log-dir/dag_1463531934548_0001_1.dot
2016-05-18 00:38:54,696 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-log-dir/dag_1463531934548_0001_1-tez-dag.pb.txt
2016-05-18 00:38:54,711 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531934548_0001_1
2016-05-18 00:38:54,711 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,711 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 00:38:54 Running Dag: dag_1463531934548_0001_1
2016-05-18 00:38:54,721 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463531934548_0001_1, submitTime=1463531934695
2016-05-18 00:38:54,726 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001/tez-conf.pb" } size: 36903 timestamp: 1463531934000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:54,726 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,726 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:54,727 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001/tez-conf.pb" } size: 36903 timestamp: 1463531934000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:38:54,727 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,727 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:38:54,727 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:38:54,728 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463531934548_0001_1, initTime=1463531934728
2016-05-18 00:38:54,728 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531934548_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:38:54,729 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:38:54,729 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:DAG_STARTED]: dagID=dag_1463531934548_0001_1, startTime=1463531934729
2016-05-18 00:38:54,729 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531934548_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:38:54,729 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E1D5268CCB064D26B3A4B78340895E89 : {D0A61EA3E6BC400D81BEA2144884FD13={InputName=D0A61EA3E6BC400D81BEA2144884FD13}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:38:54,730 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463531934548_0001, dagName=[26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 00:38:54,730 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: D0A61EA3E6BC400D81BEA2144884FD13, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:38:54,730 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,730 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,731 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,731 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,731 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:38:54,732 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,732 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: D0A61EA3E6BC400D81BEA2144884FD13 on vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,735 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input D0A61EA3E6BC400D81BEA2144884FD13 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:38:54,735 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:38:54,735 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,736 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,737 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531934548_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:38:54,737 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7], numAdditionalOutputs=1
2016-05-18 00:38:54,738 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=B728D3D5408540F0B9322DAF6997BA80, vertex=vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:38:54,738 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:38:54,738 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 00:38:54,738 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:38:54,739 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:38:54,739 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: D0A61EA3E6BC400D81BEA2144884FD13 on vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,740 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D3DD26D403FF4B8CA11953AA8171FDF7:B728D3D5408540F0B9322DAF6997BA80 using oldmapred API
2016-05-18 00:38:54,740 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=B728D3D5408540F0B9322DAF6997BA80, vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:38:54,742 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, vertexId=vertex_1463531934548_0001_1_01, initRequestedTime=1463531934732, initedTime=1463531934737, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:38:54,742 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:38:54,743 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:54,744 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {D0A61EA3E6BC400D81BEA2144884FD13=forAllWorkUnits=true, update=[1]}
2016-05-18 00:38:54,744 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] parallelism set to 1
2016-05-18 00:38:54,744 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:38:54,746 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, vertexId=vertex_1463531934548_0001_1_00, initRequestedTime=1463531934729, initedTime=1463531934745, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:38:54,746 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] with distanceFromRoot: 0
2016-05-18 00:38:54,746 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:38:54,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531934548_0001_1_00, startRequestedTime=1463531934732, startedTime=1463531934746
2016-05-18 00:38:54,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463531934548_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:38:54,747 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:54,748 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463531934548_0001_1_00 for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] numStartedSources: 1 numSources: 1
2016-05-18 00:38:54,748 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E1D5268CCB064D26B3A4B78340895E89
2016-05-18 00:38:54,748 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,748 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] with distanceFromRoot: 1
2016-05-18 00:38:54,750 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463531934548_0001_1_01, startRequestedTime=1463531934748, startedTime=1463531934749
2016-05-18 00:38:54,750 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:38:54,751 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D3DD26D403FF4B8CA11953AA8171FDF7 with 0 source tasks and 1 pending tasks
2016-05-18 00:38:54,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_STARTED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, taskId=task_1463531934548_0001_1_00_000000, scheduledTime=1463531934751, launchTime=1463531934751
2016-05-18 00:38:54,752 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,752 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E1D5268CCB064D26B3A4B78340895E89 in vertex: D3DD26D403FF4B8CA11953AA8171FDF7 numBipartiteSourceTasks: 1
2016-05-18 00:38:54,754 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:38:54,755 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463531934548_0001, with dag id: dag_1463531934548_0001_1
2016-05-18 00:38:54,756 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531934548_0001_00_000001, launchTime=1463531934755
2016-05-18 00:38:54,756 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531934548_0001_00_000001
2016-05-18 00:38:54,758 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531934548_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,758 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531934548_0001_1_00_000000_0] started. Is using containerId: [container_1463531934548_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:38:54 Starting to run new task attempt: attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,758 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0, startTime=1463531934758, containerId=container_1463531934548_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:38:54,758 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,758 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,758 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:54,759 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: E1D5268CCB064D26B3A4B78340895E89, VertexParallelism: 1, TaskAttemptID:attempt_1463531934548_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0A61EA3E6BC400D81BEA2144884FD13, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D3DD26D403FF4B8CA11953AA8171FDF7, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:38:54,763 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:54,763 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,765 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463531934548_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:38:54,766 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531934548_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:38:54,768 INFO input.MRInput (MRInput.java:initialize(440)) - D0A61EA3E6BC400D81BEA2144884FD13 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:38:54,768 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: D0A61EA3E6BC400D81BEA2144884FD13
2016-05-18 00:38:54,768 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:54,774 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E1D5268CCB064D26B3A4B78340895E89, ordinal: 0
2016-05-18 00:38:54,774 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1D5268CCB064D26B3A4B78340895E89, mem on start (mb), free: 218, total: 368, max: 672
2016-05-18 00:38:54,774 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:54,774 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:54,774 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:54,774 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:54,774 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D0A61EA3E6BC400D81BEA2144884FD13:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D3DD26D403FF4B8CA11953AA8171FDF7:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:38:54,774 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:54,775 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:54,775 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:38:54,775 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D0A61EA3E6BC400D81BEA2144884FD13:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D3DD26D403FF4B8CA11953AA8171FDF7:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:38:54,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:54,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D0A61EA3E6BC400D81BEA2144884FD13 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:54,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:54,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:54,776 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D0A61EA3E6BC400D81BEA2144884FD13
2016-05-18 00:38:54,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:54,777 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,778 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 00:38:54,778 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - D0A61EA3E6BC400D81BEA2144884FD13 initialized RecordReader from event
2016-05-18 00:38:54,780 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:54,781 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: D0A61EA3E6BC400D81BEA2144884FD13
2016-05-18 00:38:54,781 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: B53A765D4902439E8598420004F24A96
2016-05-18 00:38:54,781 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] B53A765D4902439E8598420004F24A96
2016-05-18 00:38:54,783 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D3DD26D403FF4B8CA11953AA8171FDF7 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@bf66590, comparator=cascading.tuple.hadoop.util.TupleComparator@5d85d712, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:38:54,783 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D3DD26D403FF4B8CA11953AA8171FDF7: , UsingHashComparator=false
2016-05-18 00:38:54,803 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:38:54,804 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:38:54,804 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D3DD26D403FF4B8CA11953AA8171FDF7: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:38:54,804 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D3DD26D403FF4B8CA11953AA8171FDF7: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:38:54,804 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 00:38:54,805 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E1D5268CCB064D26B3A4B78340895E89, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:38:54,808 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,808 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1D5268CCB064D26B3A4B78340895E89, mem on close (mb), free: 227, total: 358, max: 672
2016-05-18 00:38:54,809 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D3DD26D403FF4B8CA11953AA8171FDF7: Starting flush of map output
2016-05-18 00:38:54,809 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D3DD26D403FF4B8CA11953AA8171FDF7: Span0.length = 13, perItem = 15
2016-05-18 00:38:54,810 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D3DD26D403FF4B8CA11953AA8171FDF7: done sorting span=0, length=13, time=0
2016-05-18 00:38:54,811 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D3DD26D403FF4B8CA11953AA8171FDF7: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 00:38:54,815 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D3DD26D403FF4B8CA11953AA8171FDF7: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-local-dir/output/attempt_1463531934548_0001_1_00_000000_0_10049_0/file.out
2016-05-18 00:38:54,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:54 Completed running task attempt: attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,819 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:38:54,820 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531934548_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:38:54,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531934548_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463531934805, Process_Duration=3, Process_End_Time=1463531934808, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 00:38:54,821 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531934548_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,822 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,822 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:54,822 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,822 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, taskAttemptId=attempt_1463531934548_0001_1_00_000000_0, creationTime=1463531934752, allocationTime=1463531934754, startTime=1463531934758, finishTime=1463531934821, timeTaken=63, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:54,822 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531934548_0001_00_000001
2016-05-18 00:38:54,823 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531934548_0001_1_00_000000_0
2016-05-18 00:38:54,824 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_FINISHED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, taskId=task_1463531934548_0001_1_00_000000, startTime=1463531934758, finishTime=1463531934823, timeTaken=65, status=SUCCEEDED, successfulAttemptID=attempt_1463531934548_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531934805, Process_Duration=3, Process_End_Time=1463531934808, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 00:38:54,824 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:54,824 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_FINISHED]: vertexName=E1D5268CCB064D26B3A4B78340895E89, vertexId=vertex_1463531934548_0001_1_00, initRequestedTime=1463531934729, initedTime=1463531934745, startRequestedTime=1463531934732, startedTime=1463531934746, finishTime=1463531934824, timeTaken=78, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463531934805, Process_Duration=3, Process_End_Time=1463531934808, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463531934758, firstTasksToStart=[ task_1463531934548_0001_1_00_000000 ], lastTaskFinishTime=1463531934821, lastTasksToFinish=[ task_1463531934548_0001_1_00_000000 ], minTaskDuration=63, maxTaskDuration=63, avgTaskDuration=63.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531934548_0001_1_00_000000 ], longestDurationTasks=[ task_1463531934548_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:54,827 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:54,828 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531934548_0001_00_000001, stoppedTime=1463531934828, exitStatus=0
2016-05-18 00:38:54,828 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] attempt: attempt_1463531934548_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:38:54,829 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D3DD26D403FF4B8CA11953AA8171FDF7
2016-05-18 00:38:54,829 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E1D5268CCB064D26B3A4B78340895E89, destinationVertexName=D3DD26D403FF4B8CA11953AA8171FDF7. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:38:54,829 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,830 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:54,830 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:54,830 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531934548_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:38:54,830 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_STARTED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, taskId=task_1463531934548_0001_1_01_000000, scheduledTime=1463531934830, launchTime=1463531934830
2016-05-18 00:38:54,831 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,834 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463531934548_0001_00_000002, launchTime=1463531934834
2016-05-18 00:38:54,835 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531934548_0001_00_000002
2016-05-18 00:38:54,837 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531934548_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,837 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463531934548_0001_1_01_000000_0] started. Is using containerId: [container_1463531934548_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:38:54,837 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0, startTime=1463531934837, containerId=container_1463531934548_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:38:54,837 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531934548_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:38:54,838 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531934548_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:54,838 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531934548_0001_00_000001, Exiting
2016-05-18 00:38:54,838 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531934548_0001_00_000001
2016-05-18 00:38:54,838 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531934548_0001_00_000001
2016-05-18 00:38:54 Starting to run new task attempt: attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,838 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,839 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,839 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:38:54,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: D3DD26D403FF4B8CA11953AA8171FDF7, VertexParallelism: 1, TaskAttemptID:attempt_1463531934548_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E1D5268CCB064D26B3A4B78340895E89, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B728D3D5408540F0B9322DAF6997BA80, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:38:54,840 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:38:54,841 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531934548_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:54,841 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,841 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531934548_0001_00_000001 completed successfully
2016-05-18 00:38:54,844 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463531934548_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,844 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463531934548_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:38:54,845 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:38:54,848 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:38:54,853 INFO output.MROutput (MROutput.java:initialize(421)) - B728D3D5408540F0B9322DAF6997BA80: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:38:54,858 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D3DD26D403FF4B8CA11953AA8171FDF7, ordinal: 1
2016-05-18 00:38:54,858 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D3DD26D403FF4B8CA11953AA8171FDF7, mem on start (mb), free: 221, total: 358, max: 672
2016-05-18 00:38:54,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:38:54,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:38:54,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:38:54,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:38:54,858 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B728D3D5408540F0B9322DAF6997BA80:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [E1D5268CCB064D26B3A4B78340895E89:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:38:54,859 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:38:54,861 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:38:54,861 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:38:54,861 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B728D3D5408540F0B9322DAF6997BA80:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [E1D5268CCB064D26B3A4B78340895E89:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:38:54,862 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:38:54,862 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E1D5268CCB064D26B3A4B78340895E89 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:38:54,863 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E1D5268CCB064D26B3A4B78340895E89: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:38:54,863 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E1D5268CCB064D26B3A4B78340895E89: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:38:54,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:38:54,864 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E1D5268CCB064D26B3A4B78340895E89 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:38:54,864 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:38:54,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E1D5268CCB064D26B3A4B78340895E89
2016-05-18 00:38:54,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:38:54,867 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,868 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E1D5268CCB064D26B3A4B78340895E89: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:38:54,869 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463531934548_0001_1_00_000000_0_10049} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463531934869, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:38:54,869 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E1D5268CCB064D26B3A4B78340895E89
2016-05-18 00:38:54,869 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:54,870 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E1D5268CCB064D26B3A4B78340895E89, wasInterrupted=false
2016-05-18 00:38:54,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:38:54,872 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:38:54,873 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001_wd/localmode-local-dir/output/attempt_1463531934548_0001_1_00_000000_0_10049/file.out, len=175, isLocal=true
2016-05-18 00:38:54,873 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 00:38:54,873 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:38:54,873 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:38:54,875 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 00:38:54,876 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E1D5268CCB064D26B3A4B78340895E89
2016-05-18 00:38:54,876 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E1D5268CCB064D26B3A4B78340895E89: Shuffle Runner thread complete
2016-05-18 00:38:54,877 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 00:38:54,877 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: B53A765D4902439E8598420004F24A96
2016-05-18 00:38:54,877 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"], id: B728D3D5408540F0B9322DAF6997BA80
2016-05-18 00:38:54,878 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-18 00:38:54,879 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] B53A765D4902439E8598420004F24A96, for 1 inputs
2016-05-18 00:38:54,880 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D3DD26D403FF4B8CA11953AA8171FDF7, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:38:54,880 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E1D5268CCB064D26B3A4B78340895E89: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:38:54,883 INFO output.MROutput (MROutput.java:close(526)) - B728D3D5408540F0B9322DAF6997BA80 closed
2016-05-18 00:38:54,884 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463531934548_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:38:54,884 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,884 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D3DD26D403FF4B8CA11953AA8171FDF7, mem on close (mb), free: 219, total: 358, max: 672
2016-05-18 00:38:54,885 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E1D5268CCB064D26B3A4B78340895E89
2016-05-18 00:38:54,885 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E1D5268CCB064D26B3A4B78340895E89: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:38:54,885 INFO output.MROutput (MROutput.java:close(526)) - B728D3D5408540F0B9322DAF6997BA80 closed
2016-05-18 00:38:54,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:38:54 Completed running task attempt: attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,887 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:38:54,887 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463531934548_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:38:54,888 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463531934548_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463531934880, Process_Duration=4, Process_End_Time=1463531934884, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 00:38:54,889 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463531934548_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,889 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:38:54,889 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:38:54,889 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, taskAttemptId=attempt_1463531934548_0001_1_01_000000_0, creationTime=1463531934831, allocationTime=1463531934833, startTime=1463531934837, finishTime=1463531934889, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:38:54,890 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463531934548_0001_1_01_000000_0
2016-05-18 00:38:54,890 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463531934548_0001_00_000002
2016-05-18 00:38:54,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:TASK_FINISHED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, taskId=task_1463531934548_0001_1_01_000000, startTime=1463531934837, finishTime=1463531934890, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463531934548_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531934880, Process_Duration=4, Process_End_Time=1463531934884, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 00:38:54,891 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:38:54,891 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7]
2016-05-18 00:38:54,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:VERTEX_FINISHED]: vertexName=D3DD26D403FF4B8CA11953AA8171FDF7, vertexId=vertex_1463531934548_0001_1_01, initRequestedTime=1463531934732, initedTime=1463531934737, startRequestedTime=1463531934748, startedTime=1463531934749, finishTime=1463531934891, timeTaken=142, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463531934880, Process_Duration=4, Process_End_Time=1463531934884, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463531934837, firstTasksToStart=[ task_1463531934548_0001_1_01_000000 ], lastTaskFinishTime=1463531934889, lastTasksToFinish=[ task_1463531934548_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463531934548_0001_1_01_000000 ], longestDurationTasks=[ task_1463531934548_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:38:54,893 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:38:54,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463531934548_0001_00_000002, stoppedTime=1463531934893, exitStatus=0
2016-05-18 00:38:54,893 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:38:54,893 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:54,893 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463531934548_0001_1_00 [E1D5268CCB064D26B3A4B78340895E89]
2016-05-18 00:38:54,893 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463531934548_0001_1
2016-05-18 00:38:54,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463531934548_0001_1
2016-05-18 00:38:54,894 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: B728D3D5408540F0B9322DAF6997BA80 for vertex: vertex_1463531934548_0001_1_01 [D3DD26D403FF4B8CA11953AA8171FDF7], outputName: B728D3D5408540F0B9322DAF6997BA80
2016-05-18 00:38:54,894 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531934548_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:38:54,894 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463531934548_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:38:54,900 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:B728D3D5408540F0B9322DAF6997BA80 of vertex/vertexGroup:D3DD26D403FF4B8CA11953AA8171FDF7 isVertexGroupOutput:false
2016-05-18 00:38:54,900 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:38:54,900 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463531934548_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:38:54,901 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463531934548_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:38:54,901 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463531934548_0001_00_000002, Exiting
2016-05-18 00:38:54,901 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463531934548_0001_00_000002
2016-05-18 00:38:54,901 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463531934548_0001_00_000002
2016-05-18 00:38:54,902 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463531934548_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:38:54,902 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463531934548_0001_00_000002 completed successfully
2016-05-18 00:38:54,903 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463531934548_0001_1, queueSize=0
2016-05-18 00:38:54,904 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463531934548_0001_1][Event:DAG_FINISHED]: dagId=dag_1463531934548_0001_1, startTime=1463531934729, finishTime=1463531934900, timeTaken=171, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=6, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927063869685, Process_Duration=7, Process_End_Time=2927063869692, Read_Duration=1, Tuples_Read=18, Tuples_Written=18, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 00:38:54,904 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463531934548_0001_1 finished with state: SUCCEEDED
2016-05-18 00:38:54,905 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463531934548_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:38:54 Completed Dag: dag_1463531934548_0001_1
2016-05-18 00:38:54,905 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463531934548_0001_1, dagState=SUCCEEDED
2016-05-18 00:38:54,906 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463531934548_0001_1_post
2016-05-18 00:38:54,906 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:38:54,906 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:38:54,906 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:38:54,909 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn, with id=dag_1463531934548_0001_1
2016-05-18 00:38:54,909 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF] test/(1/1) ...m/testFirstNBuffer/firstn, with id=dag_1463531934548_0001_1
2016-05-18 00:38:54,913 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463531934548_0001
2016-05-18 00:38:54,918 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:38:54,918 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:38:54,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:38:54,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:38:54,918 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:38:54,918 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:38:54,919 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:38:54,921 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:38:54,921 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:38:54,921 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:38:54,922 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:38:54,922 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:38:54,922 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:38:54,923 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:38:54,926 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/26271A69689B4987823A90650E89A06B/CB2EF15ADA734EAA95E03B93DFCDE1EF/.tez/application_1463531934548_0001
2016-05-18 00:38:54,927 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn/_temporary
2016-05-18 00:38:54,927 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.402
2016-05-18 00:38:55,011 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn