2016-05-18 01:47:31,976 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-18 01:47:31,999 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.BufferPipesPlatformTest, with platform: hadoop2-tez
2016-05-18 01:47:32,023 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-18 01:47:32,546 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-18 01:47:32,828 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-18 01:47:33,089 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:33,090 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:33,184 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: C6E67D4C80404B1A9F7190D39799DA53
2016-05-18 01:47:33,416 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.103
2016-05-18 01:47:33,450 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.137
2016-05-18 01:47:33,451 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:33,451 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:33,453 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:33,623 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:33,777 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-18 01:47:33,970 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-18 01:47:33,973 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:33,973 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:33,974 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-18 01:47:33,974 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:33,975 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:33,975 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:33,976 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...stIdentityBuffer/identity
2016-05-18 01:47:33,976 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:34,006 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:34,188 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:34,196 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:34,196 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:34,252 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001 doesn't exist and is created
2016-05-18 01:47:34,336 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:34,337 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:34,338 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd
2016-05-18 01:47:34,403 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536054195_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:34,417 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:34,418 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:34,425 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:34,425 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:34,425 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:34,426 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:34,437 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 01:47:34,437 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 01:47:34,437 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:34,448 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:34,473 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:34,475 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:34,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:34,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:34,478 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:34,478 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:34,479 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:34,480 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:34,482 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:34,482 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:34,493 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:34,499 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:34,522 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:34,524 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:34,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:34,528 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-log-dir
2016-05-18 01:47:34,528 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-log-dir/history.txt.appattempt_1463536054195_0001_000000, maxErrors=10
2016-05-18 01:47:34,529 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:34,530 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536054195_0001_000000, appSubmitTime=1463536054383, launchTime=1463536054401
2016-05-18 01:47:34,535 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:34,538 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-18 01:47:34,538 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-18 01:47:34,538 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:34,593 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48328
2016-05-18 01:47:34,595 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536054195
2016-05-18 01:47:34,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536054195_0001_000000, startTime=1463536054616
2016-05-18 01:47:34,617 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:34,638 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:34,653 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:34,654 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463536054195_0001, dagName=[BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 01:47:34,862 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 01:47:34,872 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:34,932 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536054195_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-log-dir/dag_1463536054195_0001_1.dot
2016-05-18 01:47:34,938 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-log-dir/dag_1463536054195_0001_1-tez-dag.pb.txt
2016-05-18 01:47:35,014 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536054195_0001_1
2016-05-18 01:47:35,015 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:35,015 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 01:47:35 Running Dag: dag_1463536054195_0001_1
2016-05-18 01:47:35,046 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536054195_0001_1, submitTime=1463536054862
2016-05-18 01:47:35,096 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001/tez-conf.pb" } size: 40937 timestamp: 1463536054000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:35,097 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,097 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:35,098 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001/tez-conf.pb" } size: 40937 timestamp: 1463536054000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:35,099 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,100 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:35,107 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:35,110 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536054195_0001_1, initTime=1463536055109
2016-05-18 01:47:35,111 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536054195_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:35,112 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:35,115 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:DAG_STARTED]: dagID=dag_1463536054195_0001_1, startTime=1463536055115
2016-05-18 01:47:35,115 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536054195_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:35,115 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CA5B8BB5E550483EA9C255C1A84C3E32 : {95BB2E7B368941439C2ADA07E8978B34={InputName=95BB2E7B368941439C2ADA07E8978B34}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:35,116 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 95BB2E7B368941439C2ADA07E8978B34, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:35,116 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,121 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463536054195_0001, dagName=[BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity
2016-05-18 01:47:35,125 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,125 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,129 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,136 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 95BB2E7B368941439C2ADA07E8978B34 on vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,150 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:35,151 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,166 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536054195_0001, with dag id: dag_1463536054195_0001_1
2016-05-18 01:47:35,166 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 95BB2E7B368941439C2ADA07E8978B34 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:35,180 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:35,181 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,197 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,200 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536054195_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:35,200 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB], numAdditionalOutputs=1
2016-05-18 01:47:35,200 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=FD0332F410754A65916A8DCA0CE1B625, vertex=vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:35,202 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:35,203 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:35,204 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:35,205 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:35,217 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CA32A1B7B34B4A378546AD2B997003FB:FD0332F410754A65916A8DCA0CE1B625 using oldmapred API
2016-05-18 01:47:35,219 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=FD0332F410754A65916A8DCA0CE1B625, vertexName=CA32A1B7B34B4A378546AD2B997003FB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:35,231 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 95BB2E7B368941439C2ADA07E8978B34 on vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,235 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, vertexId=vertex_1463536054195_0001_1_01, initRequestedTime=1463536055151, initedTime=1463536055200, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:35,235 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:35,247 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:35,247 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {95BB2E7B368941439C2ADA07E8978B34=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:35,249 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] parallelism set to 1
2016-05-18 01:47:35,249 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:35,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, vertexId=vertex_1463536054195_0001_1_00, initRequestedTime=1463536055115, initedTime=1463536055257, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:35,257 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] with distanceFromRoot: 0
2016-05-18 01:47:35,257 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:35,258 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536054195_0001_1_00, startRequestedTime=1463536055150, startedTime=1463536055257
2016-05-18 01:47:35,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536054195_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:35,261 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:35,261 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536054195_0001_1_00 for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] numStartedSources: 1 numSources: 1
2016-05-18 01:47:35,262 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] with distanceFromRoot: 1
2016-05-18 01:47:35,262 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536054195_0001_1_01, startRequestedTime=1463536055261, startedTime=1463536055262
2016-05-18 01:47:35,262 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:35,265 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CA5B8BB5E550483EA9C255C1A84C3E32
2016-05-18 01:47:35,266 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_STARTED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, taskId=task_1463536054195_0001_1_00_000000, scheduledTime=1463536055271, launchTime=1463536055271
2016-05-18 01:47:35,273 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CA32A1B7B34B4A378546AD2B997003FB with 0 source tasks and 1 pending tasks
2016-05-18 01:47:35,278 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CA5B8BB5E550483EA9C255C1A84C3E32 in vertex: CA32A1B7B34B4A378546AD2B997003FB numBipartiteSourceTasks: 1
2016-05-18 01:47:35,297 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,355 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:35,372 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536054195_0001_00_000001, launchTime=1463536055371
2016-05-18 01:47:35,376 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536054195_0001_00_000001
2016-05-18 01:47:35,382 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536054195_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35 Starting to run new task attempt: attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,384 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,384 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536054195_0001_1_00_000000_0] started. Is using containerId: [container_1463536054195_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:35,385 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:35,386 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:35,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0, startTime=1463536055382, containerId=container_1463536054195_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:35,393 INFO counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-18 01:47:35,400 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity, VertexName: CA5B8BB5E550483EA9C255C1A84C3E32, VertexParallelism: 1, TaskAttemptID:attempt_1463536054195_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=95BB2E7B368941439C2ADA07E8978B34, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CA32A1B7B34B4A378546AD2B997003FB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:35,401 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:35,406 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,411 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536054195_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:35,412 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536054195_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:35,435 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:35,451 INFO input.MRInput (MRInput.java:initialize(440)) - 95BB2E7B368941439C2ADA07E8978B34 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:35,452 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 95BB2E7B368941439C2ADA07E8978B34
2016-05-18 01:47:35,452 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CA5B8BB5E550483EA9C255C1A84C3E32, ordinal: 0
2016-05-18 01:47:35,453 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA5B8BB5E550483EA9C255C1A84C3E32, mem on start (mb), free: 218, total: 281, max: 672
2016-05-18 01:47:35,453 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:35,454 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:35,454 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:35,455 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:35,455 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[95BB2E7B368941439C2ADA07E8978B34:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CA32A1B7B34B4A378546AD2B997003FB:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:35,461 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:35,464 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:35,464 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:35,464 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[95BB2E7B368941439C2ADA07E8978B34:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CA32A1B7B34B4A378546AD2B997003FB:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:35,465 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:35,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 95BB2E7B368941439C2ADA07E8978B34 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:35,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 95BB2E7B368941439C2ADA07E8978B34
2016-05-18 01:47:35,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:35,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:35,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:35,472 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,490 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:35,491 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 95BB2E7B368941439C2ADA07E8978B34 initialized RecordReader from event
2016-05-18 01:47:35,518 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:35,550 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 95BB2E7B368941439C2ADA07E8978B34
2016-05-18 01:47:35,550 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 8D6AC0AAD3274279AC7C0ECC10100EE1
2016-05-18 01:47:35,550 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 8D6AC0AAD3274279AC7C0ECC10100EE1
2016-05-18 01:47:35,597 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CA32A1B7B34B4A378546AD2B997003FB using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@221a28c7, comparator=cascading.tuple.hadoop.util.TupleComparator@636b324e, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:35,602 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CA32A1B7B34B4A378546AD2B997003FB: , UsingHashComparator=false
2016-05-18 01:47:35,720 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:35,720 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:35,720 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CA32A1B7B34B4A378546AD2B997003FB: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:35,722 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CA32A1B7B34B4A378546AD2B997003FB: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:35,727 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:35,729 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CA5B8BB5E550483EA9C255C1A84C3E32, all 1 inputs ready in: 00:00:00.001
2016-05-18 01:47:35,737 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,737 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA5B8BB5E550483EA9C255C1A84C3E32, mem on close (mb), free: 105, total: 281, max: 672
2016-05-18 01:47:35,737 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CA32A1B7B34B4A378546AD2B997003FB: Starting flush of map output
2016-05-18 01:47:35,738 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CA32A1B7B34B4A378546AD2B997003FB: Span0.length = 13, perItem = 15
2016-05-18 01:47:35,739 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CA32A1B7B34B4A378546AD2B997003FB: done sorting span=0, length=13, time=1
2016-05-18 01:47:35,742 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CA32A1B7B34B4A378546AD2B997003FB: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 01:47:35,747 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CA32A1B7B34B4A378546AD2B997003FB: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-local-dir/output/attempt_1463536054195_0001_1_00_000000_0_10003_0/file.out
2016-05-18 01:47:35,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:35 Completed running task attempt: attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,776 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:35,777 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536054195_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:35,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536054195_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536055729, Process_Duration=8, Process_End_Time=1463536055737, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:35,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, taskAttemptId=attempt_1463536054195_0001_1_00_000000_0, creationTime=1463536055290, allocationTime=1463536055363, startTime=1463536055382, finishTime=1463536055784, timeTaken=402, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:35,786 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536054195_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:35,788 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:35,790 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:35,790 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:35,791 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536054195_0001_00_000001
2016-05-18 01:47:35,794 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536054195_0001_1_00_000000_0
2016-05-18 01:47:35,799 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_FINISHED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, taskId=task_1463536054195_0001_1_00_000000, startTime=1463536055382, finishTime=1463536055796, timeTaken=414, status=SUCCEEDED, successfulAttemptID=attempt_1463536054195_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536055729, Process_Duration=8, Process_End_Time=1463536055737, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:35,811 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:35,811 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:35,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_FINISHED]: vertexName=CA5B8BB5E550483EA9C255C1A84C3E32, vertexId=vertex_1463536054195_0001_1_00, initRequestedTime=1463536055115, initedTime=1463536055257, startRequestedTime=1463536055150, startedTime=1463536055257, finishTime=1463536055812, timeTaken=555, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536055729, Process_Duration=8, Process_End_Time=1463536055737, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536055382, firstTasksToStart=[ task_1463536054195_0001_1_00_000000 ], lastTaskFinishTime=1463536055784, lastTasksToFinish=[ task_1463536054195_0001_1_00_000000 ], minTaskDuration=402, maxTaskDuration=402, avgTaskDuration=402.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536054195_0001_1_00_000000 ], longestDurationTasks=[ task_1463536054195_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:35,828 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:35,831 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536054195_0001_00_000001, stoppedTime=1463536055831, exitStatus=0
2016-05-18 01:47:35,832 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] attempt: attempt_1463536054195_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:35,832 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CA32A1B7B34B4A378546AD2B997003FB
2016-05-18 01:47:35,833 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:35,833 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:35,833 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CA5B8BB5E550483EA9C255C1A84C3E32, destinationVertexName=CA32A1B7B34B4A378546AD2B997003FB. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:35,834 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,834 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536054195_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:35,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_STARTED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, taskId=task_1463536054195_0001_1_01_000000, scheduledTime=1463536055835, launchTime=1463536055835
2016-05-18 01:47:35,836 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536054195_0001_00_000002, launchTime=1463536055838
2016-05-18 01:47:35,839 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536054195_0001_00_000002
2016-05-18 01:47:35,840 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536054195_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,841 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536054195_0001_1_01_000000_0] started. Is using containerId: [container_1463536054195_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:35 Starting to run new task attempt: attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,841 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0, startTime=1463536055840, containerId=container_1463536054195_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:35,841 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,842 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:35,842 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:35,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity, VertexName: CA32A1B7B34B4A378546AD2B997003FB, VertexParallelism: 1, TaskAttemptID:attempt_1463536054195_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CA5B8BB5E550483EA9C255C1A84C3E32, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FD0332F410754A65916A8DCA0CE1B625, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:35,843 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:35,844 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,853 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536054195_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:35,853 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536054195_0001_00_000001: 61 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:35,853 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536054195_0001_00_000001, Exiting
2016-05-18 01:47:35,854 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536054195_0001_00_000001
2016-05-18 01:47:35,854 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536054195_0001_00_000001
2016-05-18 01:47:35,855 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:35,860 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536054195_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:35,861 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536054195_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:35,867 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536054195_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:35,867 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536054195_0001_00_000001 completed successfully
2016-05-18 01:47:35,882 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:35,891 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CA32A1B7B34B4A378546AD2B997003FB, ordinal: 1
2016-05-18 01:47:35,891 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA32A1B7B34B4A378546AD2B997003FB, mem on start (mb), free: 92, total: 281, max: 672
2016-05-18 01:47:35,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:35,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:35,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:35,897 INFO output.MROutput (MROutput.java:initialize(421)) - FD0332F410754A65916A8DCA0CE1B625: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:35,898 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:35,898 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[FD0332F410754A65916A8DCA0CE1B625:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [CA5B8BB5E550483EA9C255C1A84C3E32:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:35,898 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:35,899 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:35,899 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:35,899 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[FD0332F410754A65916A8DCA0CE1B625:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [CA5B8BB5E550483EA9C255C1A84C3E32:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:35,899 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:35,899 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CA5B8BB5E550483EA9C255C1A84C3E32 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:35,900 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:35,900 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CA5B8BB5E550483EA9C255C1A84C3E32: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:35,900 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:35,903 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CA5B8BB5E550483EA9C255C1A84C3E32: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:35,918 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CA5B8BB5E550483EA9C255C1A84C3E32 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:35,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CA5B8BB5E550483EA9C255C1A84C3E32
2016-05-18 01:47:35,924 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:35,925 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,932 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CA5B8BB5E550483EA9C255C1A84C3E32: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:35,940 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:35,943 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536054195_0001_1_00_000000_0_10003} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463536055943, TimeTaken=4, Rate=0.04 MB/s
2016-05-18 01:47:35,943 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CA5B8BB5E550483EA9C255C1A84C3E32
2016-05-18 01:47:35,944 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:35,944 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CA5B8BB5E550483EA9C255C1A84C3E32, wasInterrupted=false
2016-05-18 01:47:35,945 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:35,948 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:35,949 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001_wd/localmode-local-dir/output/attempt_1463536054195_0001_1_00_000000_0_10003/file.out, len=175, isLocal=true
2016-05-18 01:47:35,950 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 01:47:35,951 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:35,955 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - CA5B8BB5E550483EA9C255C1A84C3E32: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:35,957 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 8D6AC0AAD3274279AC7C0ECC10100EE1
2016-05-18 01:47:35,957 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"], id: FD0332F410754A65916A8DCA0CE1B625
2016-05-18 01:47:35,958 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-18 01:47:35,958 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:35,959 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 8D6AC0AAD3274279AC7C0ECC10100EE1, for 1 inputs
2016-05-18 01:47:35,962 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 01:47:35,963 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CA32A1B7B34B4A378546AD2B997003FB, all 1 inputs ready in: 00:00:00.004
2016-05-18 01:47:35,963 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CA5B8BB5E550483EA9C255C1A84C3E32
2016-05-18 01:47:35,964 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CA5B8BB5E550483EA9C255C1A84C3E32: Shuffle Runner thread complete
2016-05-18 01:47:35,965 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CA5B8BB5E550483EA9C255C1A84C3E32: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:35,979 INFO output.MROutput (MROutput.java:close(526)) - FD0332F410754A65916A8DCA0CE1B625 closed
2016-05-18 01:47:35,980 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536054195_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:35,981 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,982 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CA32A1B7B34B4A378546AD2B997003FB, mem on close (mb), free: 85, total: 281, max: 672
2016-05-18 01:47:35,982 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CA5B8BB5E550483EA9C255C1A84C3E32
2016-05-18 01:47:35,982 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CA5B8BB5E550483EA9C255C1A84C3E32: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:35,983 INFO output.MROutput (MROutput.java:close(526)) - FD0332F410754A65916A8DCA0CE1B625 closed
2016-05-18 01:47:35,983 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:35 Completed running task attempt: attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,985 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:35,985 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536054195_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:35,986 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536054195_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=32, LAST_EVENT_RECEIVED=32][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536055963, Process_Duration=18, Process_End_Time=1463536055981, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=13]]
2016-05-18 01:47:35,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, taskAttemptId=attempt_1463536054195_0001_1_01_000000_0, creationTime=1463536055836, allocationTime=1463536055837, startTime=1463536055840, finishTime=1463536055989, timeTaken=149, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:35,989 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536054195_0001_1_01_000000_0
2016-05-18 01:47:35,988 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536054195_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:35,990 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:35,991 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:35,991 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:35,991 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536054195_0001_00_000002
2016-05-18 01:47:35,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:TASK_FINISHED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, taskId=task_1463536054195_0001_1_01_000000, startTime=1463536055840, finishTime=1463536055990, timeTaken=150, status=SUCCEEDED, successfulAttemptID=attempt_1463536054195_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=32, LAST_EVENT_RECEIVED=32, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536055963, Process_Duration=18, Process_End_Time=1463536055981, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=13
2016-05-18 01:47:35,998 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:35,998 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB]
2016-05-18 01:47:36,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:VERTEX_FINISHED]: vertexName=CA32A1B7B34B4A378546AD2B997003FB, vertexId=vertex_1463536054195_0001_1_01, initRequestedTime=1463536055151, initedTime=1463536055200, startRequestedTime=1463536055261, startedTime=1463536055262, finishTime=1463536055998, timeTaken=736, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=32, LAST_EVENT_RECEIVED=32, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536055963, Process_Duration=18, Process_End_Time=1463536055981, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=13, vertexStats=firstTaskStartTime=1463536055840, firstTasksToStart=[ task_1463536054195_0001_1_01_000000 ], lastTaskFinishTime=1463536055989, lastTasksToFinish=[ task_1463536054195_0001_1_01_000000 ], minTaskDuration=149, maxTaskDuration=149, avgTaskDuration=149.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536054195_0001_1_01_000000 ], longestDurationTasks=[ task_1463536054195_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:36,002 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:36,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536054195_0001_00_000002, stoppedTime=1463536056002, exitStatus=0
2016-05-18 01:47:36,002 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:36,002 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:36,004 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536054195_0001_1_00 [CA5B8BB5E550483EA9C255C1A84C3E32]
2016-05-18 01:47:36,004 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536054195_0001_1
2016-05-18 01:47:36,005 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536054195_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:36,005 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536054195_0001_00_000002: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:36,005 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536054195_0001_00_000002, Exiting
2016-05-18 01:47:36,005 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536054195_0001_00_000002
2016-05-18 01:47:36,006 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536054195_0001_00_000002
2016-05-18 01:47:36,007 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536054195_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:36,007 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536054195_0001_00_000002 completed successfully
2016-05-18 01:47:36,009 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536054195_0001_1
2016-05-18 01:47:36,009 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: FD0332F410754A65916A8DCA0CE1B625 for vertex: vertex_1463536054195_0001_1_01 [CA32A1B7B34B4A378546AD2B997003FB], outputName: FD0332F410754A65916A8DCA0CE1B625
2016-05-18 01:47:36,011 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536054195_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:36,011 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463536054195_0001_00_000002
2016-05-18 01:47:36,024 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:FD0332F410754A65916A8DCA0CE1B625 of vertex/vertexGroup:CA32A1B7B34B4A378546AD2B997003FB isVertexGroupOutput:false
2016-05-18 01:47:36,024 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:36,032 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536054195_0001_1, queueSize=0
2016-05-18 01:47:36,035 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536054195_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536054195_0001_1, startTime=1463536055112, finishTime=1463536056024, timeTaken=912, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=2210, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=26, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=45, MERGE_PHASE_TIME=63, FIRST_EVENT_RECEIVED=32, LAST_EVENT_RECEIVED=32, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072111692, Process_Duration=26, Process_End_Time=2927072111718, Read_Duration=4, Tuples_Read=26, Tuples_Written=26, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=13
2016-05-18 01:47:36,035 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536054195_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:36,036 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536054195_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:36 Completed Dag: dag_1463536054195_0001_1
2016-05-18 01:47:36,039 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536054195_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:36,040 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536054195_0001_1_post
2016-05-18 01:47:36,040 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:36,040 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:36,041 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:36,042 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity, with id=dag_1463536054195_0001_1
2016-05-18 01:47:36,050 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463536054195_0001
2016-05-18 01:47:36,050 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D] test/(1/1) ...stIdentityBuffer/identity, with id=dag_1463536054195_0001_1
2016-05-18 01:47:36,054 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:36,054 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:36,054 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:36,054 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:36,055 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:36,056 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:36,059 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:36,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:36,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:36,059 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:36,059 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:36,060 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:36,060 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:36,061 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:36,066 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/BD699104F6744618B4D6B4B997730BA1/EDBDBB8ACF0844AFA55A668B5916603D/.tez/application_1463536054195_0001
2016-05-18 01:47:36,068 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity/_temporary
2016-05-18 01:47:36,069 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:02.094
2016-05-18 01:47:36,196 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-18 01:47:36,249 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:36,277 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:36,357 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.032
2016-05-18 01:47:36,378 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.053
2016-05-18 01:47:36,379 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:36,380 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:36,380 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:36,412 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:36,439 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-18 01:47:36,532 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:36,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:36,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-18 01:47:36,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:36,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:36,534 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:36,534 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferForward/firstn
2016-05-18 01:47:36,535 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:36,552 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:36,558 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:36,558 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:36,558 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:36,569 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001 doesn't exist and is created
2016-05-18 01:47:36,595 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:36,596 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:36,597 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd
2016-05-18 01:47:36,614 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536056558_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:36,614 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:36,614 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:36,616 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:36,617 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:36,617 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:36,617 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:36,618 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:36,622 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:36,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:36,623 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:36,623 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:36,623 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:36,627 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:36,628 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:36,629 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:36,629 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-log-dir
2016-05-18 01:47:36,629 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-log-dir/history.txt.appattempt_1463536056558_0001_000000, maxErrors=10
2016-05-18 01:47:36,630 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:36,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536056558_0001_000000, appSubmitTime=1463536056613, launchTime=1463536056613
2016-05-18 01:47:36,640 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:36,648 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60874
2016-05-18 01:47:36,649 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536056558
2016-05-18 01:47:36,655 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536056558_0001_000000, startTime=1463536056655
2016-05-18 01:47:36,655 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:36,696 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:36,696 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:36,697 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463536056558_0001, dagName=[B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 01:47:36,719 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 01:47:36,719 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:36,721 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536056558_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-log-dir/dag_1463536056558_0001_1.dot
2016-05-18 01:47:36,722 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-log-dir/dag_1463536056558_0001_1-tez-dag.pb.txt
2016-05-18 01:47:36,730 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536056558_0001_1
2016-05-18 01:47:36,731 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:36,731 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 01:47:36 Running Dag: dag_1463536056558_0001_1
2016-05-18 01:47:36,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536056558_0001_1, submitTime=1463536056719
2016-05-18 01:47:36,749 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001/tez-conf.pb" } size: 40937 timestamp: 1463536056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:36,749 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,749 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:36,752 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001/tez-conf.pb" } size: 40937 timestamp: 1463536056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:36,752 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,752 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:36,753 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:36,754 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536056558_0001_1, initTime=1463536056754
2016-05-18 01:47:36,755 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536056558_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:36,755 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:36,756 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463536056558_0001, dagName=[B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn
2016-05-18 01:47:36,757 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:DAG_STARTED]: dagID=dag_1463536056558_0001_1, startTime=1463536056756
2016-05-18 01:47:36,757 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536056558_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:36,757 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E5D3168B31B94302BEE6594976A360EF : {45570ADDCFDE43788DC1EE85DDA7759E={InputName=45570ADDCFDE43788DC1EE85DDA7759E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:36,757 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 45570ADDCFDE43788DC1EE85DDA7759E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:36,757 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,757 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,758 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,758 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,759 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:36,759 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,762 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 45570ADDCFDE43788DC1EE85DDA7759E on vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,765 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:36,766 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,771 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,773 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536056558_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:36,773 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E], numAdditionalOutputs=1
2016-05-18 01:47:36,773 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=CBDCCB2EF5AA41F6A3CF1454D3A8C5B6, vertex=vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:36,774 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 45570ADDCFDE43788DC1EE85DDA7759E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:36,777 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for EDD90434A6154EF597D162B7BE85833E:CBDCCB2EF5AA41F6A3CF1454D3A8C5B6 using oldmapred API
2016-05-18 01:47:36,779 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=CBDCCB2EF5AA41F6A3CF1454D3A8C5B6, vertexName=EDD90434A6154EF597D162B7BE85833E, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:36,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EDD90434A6154EF597D162B7BE85833E, vertexId=vertex_1463536056558_0001_1_01, initRequestedTime=1463536056759, initedTime=1463536056773, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:36,781 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:36,782 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:36,782 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:36,782 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:36,782 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:36,784 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 45570ADDCFDE43788DC1EE85DDA7759E on vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,787 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:36,787 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {45570ADDCFDE43788DC1EE85DDA7759E=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:36,788 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] parallelism set to 1
2016-05-18 01:47:36,789 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:36,790 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E5D3168B31B94302BEE6594976A360EF, vertexId=vertex_1463536056558_0001_1_00, initRequestedTime=1463536056757, initedTime=1463536056789, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:36,790 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] with distanceFromRoot: 0
2016-05-18 01:47:36,791 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:36,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536056558_0001_1_00, startRequestedTime=1463536056759, startedTime=1463536056791
2016-05-18 01:47:36,793 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536056558_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:36,793 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:36,793 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E5D3168B31B94302BEE6594976A360EF
2016-05-18 01:47:36,794 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,794 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536056558_0001_1_00 for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] numStartedSources: 1 numSources: 1
2016-05-18 01:47:36,794 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] with distanceFromRoot: 1
2016-05-18 01:47:36,795 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536056558_0001_1_01, startRequestedTime=1463536056793, startedTime=1463536056794
2016-05-18 01:47:36,795 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:36,796 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536056558_0001, with dag id: dag_1463536056558_0001_1
2016-05-18 01:47:36,798 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_STARTED]: vertexName=E5D3168B31B94302BEE6594976A360EF, taskId=task_1463536056558_0001_1_00_000000, scheduledTime=1463536056796, launchTime=1463536056796
2016-05-18 01:47:36,799 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,800 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EDD90434A6154EF597D162B7BE85833E with 0 source tasks and 1 pending tasks
2016-05-18 01:47:36,800 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:36,801 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E5D3168B31B94302BEE6594976A360EF in vertex: EDD90434A6154EF597D162B7BE85833E numBipartiteSourceTasks: 1
2016-05-18 01:47:36,804 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536056558_0001_00_000001, launchTime=1463536056804
2016-05-18 01:47:36,805 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536056558_0001_00_000001
2016-05-18 01:47:36,808 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536056558_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,808 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536056558_0001_1_00_000000_0] started. Is using containerId: [container_1463536056558_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:36,808 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E5D3168B31B94302BEE6594976A360EF, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0, startTime=1463536056807, containerId=container_1463536056558_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:36 Starting to run new task attempt: attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,809 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,809 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:36,809 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:36,809 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn, VertexName: E5D3168B31B94302BEE6594976A360EF, VertexParallelism: 1, TaskAttemptID:attempt_1463536056558_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=45570ADDCFDE43788DC1EE85DDA7759E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EDD90434A6154EF597D162B7BE85833E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:36,810 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:36,811 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,816 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536056558_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:36,816 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536056558_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:36,820 INFO input.MRInput (MRInput.java:initialize(440)) - 45570ADDCFDE43788DC1EE85DDA7759E using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:36,820 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 45570ADDCFDE43788DC1EE85DDA7759E
2016-05-18 01:47:36,820 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:36,828 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E5D3168B31B94302BEE6594976A360EF, ordinal: 0
2016-05-18 01:47:36,828 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5D3168B31B94302BEE6594976A360EF, mem on start (mb), free: 76, total: 281, max: 672
2016-05-18 01:47:36,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:36,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:36,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:36,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:36,830 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[45570ADDCFDE43788DC1EE85DDA7759E:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [EDD90434A6154EF597D162B7BE85833E:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:36,830 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:36,830 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:36,831 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:36,831 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[45570ADDCFDE43788DC1EE85DDA7759E:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [EDD90434A6154EF597D162B7BE85833E:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:36,831 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:36,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 45570ADDCFDE43788DC1EE85DDA7759E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:36,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 45570ADDCFDE43788DC1EE85DDA7759E
2016-05-18 01:47:36,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:36,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:36,833 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:36,833 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,835 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:36,835 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 45570ADDCFDE43788DC1EE85DDA7759E initialized RecordReader from event
2016-05-18 01:47:36,841 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:36,843 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 45570ADDCFDE43788DC1EE85DDA7759E
2016-05-18 01:47:36,843 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: B79DBAEE0EB6445D996FA2F2BDD599BF
2016-05-18 01:47:36,843 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] B79DBAEE0EB6445D996FA2F2BDD599BF
2016-05-18 01:47:36,849 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EDD90434A6154EF597D162B7BE85833E using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@676d9b56, comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@4cf16026, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:36,850 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EDD90434A6154EF597D162B7BE85833E: , UsingHashComparator=false
2016-05-18 01:47:36,954 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:36,954 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:36,955 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EDD90434A6154EF597D162B7BE85833E: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:36,956 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EDD90434A6154EF597D162B7BE85833E: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:36,957 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:36,957 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E5D3168B31B94302BEE6594976A360EF, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:36,961 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,961 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E5D3168B31B94302BEE6594976A360EF, mem on close (mb), free: 73, total: 382, max: 672
2016-05-18 01:47:36,962 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EDD90434A6154EF597D162B7BE85833E: Starting flush of map output
2016-05-18 01:47:36,962 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EDD90434A6154EF597D162B7BE85833E: Span0.length = 13, perItem = 22
2016-05-18 01:47:36,963 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EDD90434A6154EF597D162B7BE85833E: done sorting span=0, length=13, time=1
2016-05-18 01:47:36,963 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EDD90434A6154EF597D162B7BE85833E: Heap = SpanIterator<0:12> (span=Span[16777216,286]),
2016-05-18 01:47:36,965 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EDD90434A6154EF597D162B7BE85833E: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-local-dir/output/attempt_1463536056558_0001_1_00_000000_0_10009_0/file.out
2016-05-18 01:47:36,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:36 Completed running task attempt: attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,972 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:36,972 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536056558_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:36,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536056558_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536056957, Process_Duration=4, Process_End_Time=1463536056961, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:36,974 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536056558_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:36,974 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:36,974 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:36,974 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:36,975 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E5D3168B31B94302BEE6594976A360EF, taskAttemptId=attempt_1463536056558_0001_1_00_000000_0, creationTime=1463536056798, allocationTime=1463536056803, startTime=1463536056807, finishTime=1463536056974, timeTaken=167, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:36,976 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536056558_0001_1_00_000000_0
2016-05-18 01:47:36,977 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_FINISHED]: vertexName=E5D3168B31B94302BEE6594976A360EF, taskId=task_1463536056558_0001_1_00_000000, startTime=1463536056807, finishTime=1463536056976, timeTaken=169, status=SUCCEEDED, successfulAttemptID=attempt_1463536056558_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536056957, Process_Duration=4, Process_End_Time=1463536056961, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:36,978 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536056558_0001_00_000001
2016-05-18 01:47:36,978 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:36,979 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:36,981 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_FINISHED]: vertexName=E5D3168B31B94302BEE6594976A360EF, vertexId=vertex_1463536056558_0001_1_00, initRequestedTime=1463536056757, initedTime=1463536056789, startRequestedTime=1463536056759, startedTime=1463536056791, finishTime=1463536056979, timeTaken=188, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536056957, Process_Duration=4, Process_End_Time=1463536056961, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536056807, firstTasksToStart=[ task_1463536056558_0001_1_00_000000 ], lastTaskFinishTime=1463536056974, lastTasksToFinish=[ task_1463536056558_0001_1_00_000000 ], minTaskDuration=167, maxTaskDuration=167, avgTaskDuration=167.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536056558_0001_1_00_000000 ], longestDurationTasks=[ task_1463536056558_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:36,981 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:36,981 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] attempt: attempt_1463536056558_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:36,982 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536056558_0001_00_000001, stoppedTime=1463536056982, exitStatus=0
2016-05-18 01:47:36,982 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:36,982 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:36,982 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EDD90434A6154EF597D162B7BE85833E
2016-05-18 01:47:36,983 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536056558_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:36,983 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E5D3168B31B94302BEE6594976A360EF, destinationVertexName=EDD90434A6154EF597D162B7BE85833E. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:36,983 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,984 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_STARTED]: vertexName=EDD90434A6154EF597D162B7BE85833E, taskId=task_1463536056558_0001_1_01_000000, scheduledTime=1463536056984, launchTime=1463536056984
2016-05-18 01:47:36,984 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:36,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536056558_0001_00_000002, launchTime=1463536056986
2016-05-18 01:47:36,992 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536056558_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:36,992 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536056558_0001_00_000001: 13 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:36,992 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536056558_0001_00_000001, Exiting
2016-05-18 01:47:36,992 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536056558_0001_00_000001
2016-05-18 01:47:36,992 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536056558_0001_00_000001
2016-05-18 01:47:36,994 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536056558_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:36,995 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536056558_0001_00_000001 completed successfully
2016-05-18 01:47:36,995 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536056558_0001_00_000002
2016-05-18 01:47:36,996 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536056558_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:36,996 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536056558_0001_1_01_000000_0] started. Is using containerId: [container_1463536056558_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:36 Starting to run new task attempt: attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:36,997 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:36,997 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:36,997 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:36,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EDD90434A6154EF597D162B7BE85833E, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0, startTime=1463536056996, containerId=container_1463536056558_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:36,997 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn, VertexName: EDD90434A6154EF597D162B7BE85833E, VertexParallelism: 1, TaskAttemptID:attempt_1463536056558_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E5D3168B31B94302BEE6594976A360EF, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=CBDCCB2EF5AA41F6A3CF1454D3A8C5B6, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:36,998 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:36,999 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:37,000 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536056558_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:37,000 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536056558_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:37,008 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:37,009 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:37,023 INFO output.MROutput (MROutput.java:initialize(421)) - CBDCCB2EF5AA41F6A3CF1454D3A8C5B6: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:37,027 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EDD90434A6154EF597D162B7BE85833E, ordinal: 1
2016-05-18 01:47:37,027 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDD90434A6154EF597D162B7BE85833E, mem on start (mb), free: 66, total: 382, max: 672
2016-05-18 01:47:37,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:37,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:37,028 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:37,029 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:37,030 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CBDCCB2EF5AA41F6A3CF1454D3A8C5B6:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [E5D3168B31B94302BEE6594976A360EF:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:37,030 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:37,031 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:37,032 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:37,032 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CBDCCB2EF5AA41F6A3CF1454D3A8C5B6:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [E5D3168B31B94302BEE6594976A360EF:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:37,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:37,035 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E5D3168B31B94302BEE6594976A360EF: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:37,035 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E5D3168B31B94302BEE6594976A360EF: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:37,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E5D3168B31B94302BEE6594976A360EF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:37,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:37,036 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:37,038 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E5D3168B31B94302BEE6594976A360EF with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:37,040 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E5D3168B31B94302BEE6594976A360EF
2016-05-18 01:47:37,040 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:37,041 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:37,041 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E5D3168B31B94302BEE6594976A360EF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:37,045 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536056558_0001_1_00_000000_0_10009} to DISK_DIRECT, csize=322, dsize=318, EndTime=1463536057045, TimeTaken=3, Rate=0.10 MB/s
2016-05-18 01:47:37,045 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E5D3168B31B94302BEE6594976A360EF
2016-05-18 01:47:37,045 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:37,046 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E5D3168B31B94302BEE6594976A360EF, wasInterrupted=false
2016-05-18 01:47:37,048 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:37,048 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:37,048 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - E5D3168B31B94302BEE6594976A360EF: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:37,050 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: B79DBAEE0EB6445D996FA2F2BDD599BF
2016-05-18 01:47:37,050 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:37,051 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"], id: CBDCCB2EF5AA41F6A3CF1454D3A8C5B6
2016-05-18 01:47:37,051 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-18 01:47:37,051 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001_wd/localmode-local-dir/output/attempt_1463536056558_0001_1_00_000000_0_10009/file.out, len=322, isLocal=true
2016-05-18 01:47:37,052 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 322 bytes from disk
2016-05-18 01:47:37,052 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:37,052 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:37,052 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] B79DBAEE0EB6445D996FA2F2BDD599BF, for 1 inputs
2016-05-18 01:47:37,053 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-18 01:47:37,053 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E5D3168B31B94302BEE6594976A360EF
2016-05-18 01:47:37,053 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E5D3168B31B94302BEE6594976A360EF: Shuffle Runner thread complete
2016-05-18 01:47:37,054 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EDD90434A6154EF597D162B7BE85833E, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:37,055 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E5D3168B31B94302BEE6594976A360EF: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.GroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:37,061 INFO output.MROutput (MROutput.java:close(526)) - CBDCCB2EF5AA41F6A3CF1454D3A8C5B6 closed
2016-05-18 01:47:37,062 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536056558_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:37,063 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:37,063 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDD90434A6154EF597D162B7BE85833E, mem on close (mb), free: 62, total: 382, max: 672
2016-05-18 01:47:37,064 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E5D3168B31B94302BEE6594976A360EF
2016-05-18 01:47:37,064 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E5D3168B31B94302BEE6594976A360EF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:37,064 INFO output.MROutput (MROutput.java:close(526)) - CBDCCB2EF5AA41F6A3CF1454D3A8C5B6 closed
2016-05-18 01:47:37,065 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:37 Completed running task attempt: attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:37,066 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:37,066 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536056558_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:37,067 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536056558_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536057054, Process_Duration=9, Process_End_Time=1463536057063, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 01:47:37,068 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536056558_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EDD90434A6154EF597D162B7BE85833E, taskAttemptId=attempt_1463536056558_0001_1_01_000000_0, creationTime=1463536056984, allocationTime=1463536056985, startTime=1463536056996, finishTime=1463536057068, timeTaken=72, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:37,069 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536056558_0001_1_01_000000_0
2016-05-18 01:47:37,068 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,070 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:37,070 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,070 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536056558_0001_00_000002
2016-05-18 01:47:37,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:TASK_FINISHED]: vertexName=EDD90434A6154EF597D162B7BE85833E, taskId=task_1463536056558_0001_1_01_000000, startTime=1463536056996, finishTime=1463536057069, timeTaken=73, status=SUCCEEDED, successfulAttemptID=attempt_1463536056558_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536057054, Process_Duration=9, Process_End_Time=1463536057063, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 01:47:37,072 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:37,072 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E]
2016-05-18 01:47:37,075 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:VERTEX_FINISHED]: vertexName=EDD90434A6154EF597D162B7BE85833E, vertexId=vertex_1463536056558_0001_1_01, initRequestedTime=1463536056759, initedTime=1463536056773, startRequestedTime=1463536056793, startedTime=1463536056794, finishTime=1463536057072, timeTaken=278, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536057054, Process_Duration=9, Process_End_Time=1463536057063, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463536056996, firstTasksToStart=[ task_1463536056558_0001_1_01_000000 ], lastTaskFinishTime=1463536057068, lastTasksToFinish=[ task_1463536056558_0001_1_01_000000 ], minTaskDuration=72, maxTaskDuration=72, avgTaskDuration=72.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536056558_0001_1_01_000000 ], longestDurationTasks=[ task_1463536056558_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:37,076 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:37,076 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536056558_0001_00_000002, stoppedTime=1463536057076, exitStatus=0
2016-05-18 01:47:37,077 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:37,077 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:37,078 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536056558_0001_1_00 [E5D3168B31B94302BEE6594976A360EF]
2016-05-18 01:47:37,078 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536056558_0001_1
2016-05-18 01:47:37,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536056558_0001_1
2016-05-18 01:47:37,079 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: CBDCCB2EF5AA41F6A3CF1454D3A8C5B6 for vertex: vertex_1463536056558_0001_1_01 [EDD90434A6154EF597D162B7BE85833E], outputName: CBDCCB2EF5AA41F6A3CF1454D3A8C5B6
2016-05-18 01:47:37,079 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536056558_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:37,080 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536056558_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:37,081 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536056558_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:37,083 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536056558_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:37,083 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536056558_0001_00_000002, Exiting
2016-05-18 01:47:37,083 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536056558_0001_00_000002
2016-05-18 01:47:37,083 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536056558_0001_00_000002
2016-05-18 01:47:37,084 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536056558_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:37,085 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536056558_0001_00_000002 completed successfully
2016-05-18 01:47:37,096 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:CBDCCB2EF5AA41F6A3CF1454D3A8C5B6 of vertex/vertexGroup:EDD90434A6154EF597D162B7BE85833E isVertexGroupOutput:false
2016-05-18 01:47:37,096 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:37,099 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536056558_0001_1, queueSize=0
2016-05-18 01:47:37,100 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536056558_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536056558_0001_1, startTime=1463536056755, finishTime=1463536057096, timeTaken=341, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=640, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=18, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072114011, Process_Duration=13, Process_End_Time=2927072114024, Read_Duration=2, Tuples_Read=18, Tuples_Written=18, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 01:47:37,100 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536056558_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:37,100 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536056558_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:37 Completed Dag: dag_1463536056558_0001_1
2016-05-18 01:47:37,101 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536056558_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:37,101 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536056558_0001_1_post
2016-05-18 01:47:37,102 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,102 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:37,102 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:37,102 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn, with id=dag_1463536056558_0001_1
2016-05-18 01:47:37,103 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989] test/(1/1) ...irstNBufferForward/firstn, with id=dag_1463536056558_0001_1
2016-05-18 01:47:37,114 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463536056558_0001
2016-05-18 01:47:37,121 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:37,121 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:37,121 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:37,121 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:37,121 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:37,122 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:37,123 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:37,123 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:37,124 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:37,124 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:37,125 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:37,126 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:37,126 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:37,128 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:37,132 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B34E2E984D3B4ECF8FFC244CC94A1A97/F0CF85B42BAC4EB8B56EF9D962E9B989/.tez/application_1463536056558_0001
2016-05-18 01:47:37,133 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn/_temporary
2016-05-18 01:47:37,134 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.600
2016-05-18 01:47:37,248 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-18 01:47:37,285 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:37,286 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:37,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.031
2016-05-18 01:47:37,385 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.063
2016-05-18 01:47:37,386 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:37,386 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:37,387 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:37,413 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:37,424 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-18 01:47:37,492 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:37,492 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:37,493 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-18 01:47:37,493 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:37,493 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:37,493 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:37,494 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testSimpleBuffer/simple
2016-05-18 01:47:37,495 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:37,511 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:37,516 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:37,516 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:37,517 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:37,526 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001 doesn't exist and is created
2016-05-18 01:47:37,556 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:37,556 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:37,558 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd
2016-05-18 01:47:37,576 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536057516_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:37,577 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:37,577 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:37,580 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:37,580 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:37,580 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:37,580 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:37,582 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:37,584 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:37,585 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:37,585 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:37,585 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:37,585 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:37,586 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:37,586 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:37,586 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:37,586 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:37,587 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:37,587 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:37,588 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:37,590 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:37,591 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:37,592 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:37,592 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-log-dir
2016-05-18 01:47:37,593 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-log-dir/history.txt.appattempt_1463536057516_0001_000000, maxErrors=10
2016-05-18 01:47:37,593 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:37,594 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536057516_0001_000000, appSubmitTime=1463536057576, launchTime=1463536057576
2016-05-18 01:47:37,602 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:37,608 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:39965
2016-05-18 01:47:37,610 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536057516
2016-05-18 01:47:37,612 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536057516_0001_000000, startTime=1463536057612
2016-05-18 01:47:37,612 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:37,657 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:37,657 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:37,658 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463536057516_0001, dagName=[74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 01:47:37,679 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 01:47:37,679 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:37,680 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536057516_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-log-dir/dag_1463536057516_0001_1.dot
2016-05-18 01:47:37,682 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-log-dir/dag_1463536057516_0001_1-tez-dag.pb.txt
2016-05-18 01:47:37,691 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536057516_0001_1
2016-05-18 01:47:37,691 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,691 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 01:47:37 Running Dag: dag_1463536057516_0001_1
2016-05-18 01:47:37,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536057516_0001_1, submitTime=1463536057679
2016-05-18 01:47:37,707 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001/tez-conf.pb" } size: 40937 timestamp: 1463536057000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:37,708 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,708 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:37,709 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001/tez-conf.pb" } size: 40937 timestamp: 1463536057000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:37,709 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,710 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:37,710 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:37,711 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536057516_0001_1, initTime=1463536057710
2016-05-18 01:47:37,711 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536057516_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:37,711 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:37,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:DAG_STARTED]: dagID=dag_1463536057516_0001_1, startTime=1463536057712
2016-05-18 01:47:37,712 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536057516_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:37,712 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C66A159D846D48C0873963CD5ECB2C83 : {B800AB581CA14631811511838E715132={InputName=B800AB581CA14631811511838E715132}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:37,712 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: B800AB581CA14631811511838E715132, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:37,712 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,713 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,713 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,713 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,713 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463536057516_0001, dagName=[74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-18 01:47:37,714 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:37,714 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: B800AB581CA14631811511838E715132 on vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,714 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,719 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input B800AB581CA14631811511838E715132 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:37,719 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:37,719 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,720 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536057516_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:37,723 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688], numAdditionalOutputs=1
2016-05-18 01:47:37,723 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=BA7DEDE812D947EA94DE91282F1633C8, vertex=vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:37,725 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FF5AA71693964EC38004075268FB4688:BA7DEDE812D947EA94DE91282F1633C8 using oldmapred API
2016-05-18 01:47:37,726 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=BA7DEDE812D947EA94DE91282F1633C8, vertexName=FF5AA71693964EC38004075268FB4688, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:37,727 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FF5AA71693964EC38004075268FB4688, vertexId=vertex_1463536057516_0001_1_01, initRequestedTime=1463536057714, initedTime=1463536057723, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:37,727 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:37,726 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:37,729 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:37,729 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:37,729 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:37,731 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: B800AB581CA14631811511838E715132 on vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,733 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:37,734 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {B800AB581CA14631811511838E715132=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:37,734 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] parallelism set to 1
2016-05-18 01:47:37,734 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:37,738 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, vertexId=vertex_1463536057516_0001_1_00, initRequestedTime=1463536057712, initedTime=1463536057737, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:37,738 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] with distanceFromRoot: 0
2016-05-18 01:47:37,738 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:37,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536057516_0001_1_00, startRequestedTime=1463536057714, startedTime=1463536057739
2016-05-18 01:47:37,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536057516_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:37,740 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:37,740 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536057516_0001_1_00 for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] numStartedSources: 1 numSources: 1
2016-05-18 01:47:37,741 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] with distanceFromRoot: 1
2016-05-18 01:47:37,740 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C66A159D846D48C0873963CD5ECB2C83
2016-05-18 01:47:37,742 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,742 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536057516_0001_1_01, startRequestedTime=1463536057740, startedTime=1463536057741
2016-05-18 01:47:37,744 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:37,745 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FF5AA71693964EC38004075268FB4688 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:37,746 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_STARTED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, taskId=task_1463536057516_0001_1_00_000000, scheduledTime=1463536057746, launchTime=1463536057746
2016-05-18 01:47:37,746 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,747 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C66A159D846D48C0873963CD5ECB2C83 in vertex: FF5AA71693964EC38004075268FB4688 numBipartiteSourceTasks: 1
2016-05-18 01:47:37,748 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:37,750 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536057516_0001_00_000001, launchTime=1463536057750
2016-05-18 01:47:37,750 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536057516_0001_00_000001
2016-05-18 01:47:37,751 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536057516_0001_1_00_000000_0] started. Is using containerId: [container_1463536057516_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:37,752 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0, startTime=1463536057751, containerId=container_1463536057516_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:37,752 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536057516_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37 Starting to run new task attempt: attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,753 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,753 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,753 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:37,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: C66A159D846D48C0873963CD5ECB2C83, VertexParallelism: 1, TaskAttemptID:attempt_1463536057516_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B800AB581CA14631811511838E715132, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FF5AA71693964EC38004075268FB4688, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:37,754 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:37,754 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,760 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536057516_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:37,760 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536057516_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:37,765 INFO input.MRInput (MRInput.java:initialize(440)) - B800AB581CA14631811511838E715132 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:37,765 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: B800AB581CA14631811511838E715132
2016-05-18 01:47:37,766 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:37,769 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536057516_0001, with dag id: dag_1463536057516_0001_1
2016-05-18 01:47:37,778 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C66A159D846D48C0873963CD5ECB2C83, ordinal: 0
2016-05-18 01:47:37,778 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C66A159D846D48C0873963CD5ECB2C83, mem on start (mb), free: 163, total: 489, max: 672
2016-05-18 01:47:37,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:37,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:37,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:37,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:37,779 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B800AB581CA14631811511838E715132:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FF5AA71693964EC38004075268FB4688:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:37,779 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:37,779 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:37,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:37,780 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B800AB581CA14631811511838E715132:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FF5AA71693964EC38004075268FB4688:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B800AB581CA14631811511838E715132 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B800AB581CA14631811511838E715132
2016-05-18 01:47:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:37,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:37,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:37,781 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,783 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:37,783 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - B800AB581CA14631811511838E715132 initialized RecordReader from event
2016-05-18 01:47:37,785 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:37,785 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: B800AB581CA14631811511838E715132
2016-05-18 01:47:37,786 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: C53CA231DE8B46209AAB4584DAE3EF79
2016-05-18 01:47:37,786 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] C53CA231DE8B46209AAB4584DAE3EF79
2016-05-18 01:47:37,789 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FF5AA71693964EC38004075268FB4688 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@43017c6e, comparator=cascading.tuple.hadoop.util.TupleComparator@311fa52c, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:37,792 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FF5AA71693964EC38004075268FB4688: , UsingHashComparator=false
2016-05-18 01:47:37,876 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:37,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:37,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FF5AA71693964EC38004075268FB4688: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:37,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FF5AA71693964EC38004075268FB4688: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:37,878 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:37,880 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C66A159D846D48C0873963CD5ECB2C83, all 1 inputs ready in: 00:00:00.001
2016-05-18 01:47:37,882 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,882 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C66A159D846D48C0873963CD5ECB2C83, mem on close (mb), free: 60, total: 489, max: 672
2016-05-18 01:47:37,883 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FF5AA71693964EC38004075268FB4688: Starting flush of map output
2016-05-18 01:47:37,883 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FF5AA71693964EC38004075268FB4688: Span0.length = 13, perItem = 15
2016-05-18 01:47:37,884 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FF5AA71693964EC38004075268FB4688: done sorting span=0, length=13, time=1
2016-05-18 01:47:37,885 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FF5AA71693964EC38004075268FB4688: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 01:47:37,886 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FF5AA71693964EC38004075268FB4688: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-local-dir/output/attempt_1463536057516_0001_1_00_000000_0_10015_0/file.out
2016-05-18 01:47:37,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:37 Completed running task attempt: attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,890 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:37,890 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536057516_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:37,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536057516_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536057880, Process_Duration=2, Process_End_Time=1463536057882, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:37,892 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536057516_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, taskAttemptId=attempt_1463536057516_0001_1_00_000000_0, creationTime=1463536057746, allocationTime=1463536057749, startTime=1463536057751, finishTime=1463536057892, timeTaken=141, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:37,893 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,893 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:37,893 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536057516_0001_1_00_000000_0
2016-05-18 01:47:37,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_FINISHED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, taskId=task_1463536057516_0001_1_00_000000, startTime=1463536057751, finishTime=1463536057894, timeTaken=143, status=SUCCEEDED, successfulAttemptID=attempt_1463536057516_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536057880, Process_Duration=2, Process_End_Time=1463536057882, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:37,894 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,895 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:37,896 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_FINISHED]: vertexName=C66A159D846D48C0873963CD5ECB2C83, vertexId=vertex_1463536057516_0001_1_00, initRequestedTime=1463536057712, initedTime=1463536057737, startRequestedTime=1463536057714, startedTime=1463536057739, finishTime=1463536057896, timeTaken=157, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536057880, Process_Duration=2, Process_End_Time=1463536057882, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536057751, firstTasksToStart=[ task_1463536057516_0001_1_00_000000 ], lastTaskFinishTime=1463536057892, lastTasksToFinish=[ task_1463536057516_0001_1_00_000000 ], minTaskDuration=141, maxTaskDuration=141, avgTaskDuration=141.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536057516_0001_1_00_000000 ], longestDurationTasks=[ task_1463536057516_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:37,897 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:37,898 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536057516_0001_00_000001, stoppedTime=1463536057898, exitStatus=0
2016-05-18 01:47:37,898 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] attempt: attempt_1463536057516_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:37,898 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FF5AA71693964EC38004075268FB4688
2016-05-18 01:47:37,899 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C66A159D846D48C0873963CD5ECB2C83, destinationVertexName=FF5AA71693964EC38004075268FB4688. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:37,899 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536057516_0001_00_000001
2016-05-18 01:47:37,899 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:37,900 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536057516_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:37,900 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:37,899 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,900 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536057516_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:37,901 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536057516_0001_00_000001, Exiting
2016-05-18 01:47:37,901 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536057516_0001_00_000001
2016-05-18 01:47:37,901 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536057516_0001_00_000001
2016-05-18 01:47:37,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_STARTED]: vertexName=FF5AA71693964EC38004075268FB4688, taskId=task_1463536057516_0001_1_01_000000, scheduledTime=1463536057901, launchTime=1463536057901
2016-05-18 01:47:37,902 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536057516_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:37,902 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,903 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536057516_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:37,904 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536057516_0001_00_000001 completed successfully
2016-05-18 01:47:37,906 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536057516_0001_00_000002, launchTime=1463536057905
2016-05-18 01:47:37,907 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536057516_0001_00_000002
2016-05-18 01:47:37,908 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536057516_0001_1_01_000000_0] started. Is using containerId: [container_1463536057516_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:37,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FF5AA71693964EC38004075268FB4688, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0, startTime=1463536057907, containerId=container_1463536057516_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:37,908 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536057516_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37 Starting to run new task attempt: attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,909 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,910 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,911 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:37,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: FF5AA71693964EC38004075268FB4688, VertexParallelism: 1, TaskAttemptID:attempt_1463536057516_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C66A159D846D48C0873963CD5ECB2C83, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=BA7DEDE812D947EA94DE91282F1633C8, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:37,911 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:37,911 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,914 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536057516_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,914 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536057516_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:37,917 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:37,924 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:37,931 INFO output.MROutput (MROutput.java:initialize(421)) - BA7DEDE812D947EA94DE91282F1633C8: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:37,936 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FF5AA71693964EC38004075268FB4688, ordinal: 1
2016-05-18 01:47:37,936 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FF5AA71693964EC38004075268FB4688, mem on start (mb), free: 53, total: 489, max: 672
2016-05-18 01:47:37,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:37,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:37,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:37,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:37,936 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BA7DEDE812D947EA94DE91282F1633C8:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C66A159D846D48C0873963CD5ECB2C83:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:37,937 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:37,938 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:37,939 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:37,939 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BA7DEDE812D947EA94DE91282F1633C8:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C66A159D846D48C0873963CD5ECB2C83:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:37,939 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:37,939 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C66A159D846D48C0873963CD5ECB2C83 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:37,940 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C66A159D846D48C0873963CD5ECB2C83: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:37,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:37,941 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:37,941 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C66A159D846D48C0873963CD5ECB2C83: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:37,942 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C66A159D846D48C0873963CD5ECB2C83 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:37,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C66A159D846D48C0873963CD5ECB2C83
2016-05-18 01:47:37,943 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:37,944 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,944 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C66A159D846D48C0873963CD5ECB2C83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:37,950 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536057516_0001_1_00_000000_0_10015} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463536057950, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 01:47:37,950 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C66A159D846D48C0873963CD5ECB2C83
2016-05-18 01:47:37,951 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:37,951 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:37,952 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C66A159D846D48C0873963CD5ECB2C83, wasInterrupted=false
2016-05-18 01:47:37,953 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: C53CA231DE8B46209AAB4584DAE3EF79
2016-05-18 01:47:37,954 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"], id: BA7DEDE812D947EA94DE91282F1633C8
2016-05-18 01:47:37,954 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-18 01:47:37,955 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:37,955 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] C53CA231DE8B46209AAB4584DAE3EF79, for 1 inputs
2016-05-18 01:47:37,955 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:37,958 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001_wd/localmode-local-dir/output/attempt_1463536057516_0001_1_00_000000_0_10015/file.out, len=175, isLocal=true
2016-05-18 01:47:37,958 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 01:47:37,959 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:37,959 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:37,960 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 01:47:37,960 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C66A159D846D48C0873963CD5ECB2C83
2016-05-18 01:47:37,960 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FF5AA71693964EC38004075268FB4688, all 1 inputs ready in: 00:00:00.005
2016-05-18 01:47:37,963 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C66A159D846D48C0873963CD5ECB2C83: Shuffle Runner thread complete
2016-05-18 01:47:37,964 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C66A159D846D48C0873963CD5ECB2C83: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:37,970 INFO output.MROutput (MROutput.java:close(526)) - BA7DEDE812D947EA94DE91282F1633C8 closed
2016-05-18 01:47:37,972 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536057516_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:37,973 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,974 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FF5AA71693964EC38004075268FB4688, mem on close (mb), free: 48, total: 489, max: 672
2016-05-18 01:47:37,974 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C66A159D846D48C0873963CD5ECB2C83
2016-05-18 01:47:37,974 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C66A159D846D48C0873963CD5ECB2C83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:37,975 INFO output.MROutput (MROutput.java:close(526)) - BA7DEDE812D947EA94DE91282F1633C8 closed
2016-05-18 01:47:37,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:37 Completed running task attempt: attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,977 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:37,978 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536057516_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:37,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536057516_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536057963, Process_Duration=10, Process_End_Time=1463536057973, Read_Duration=2, Tuples_Read=13, Tuples_Written=23, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=23]]
2016-05-18 01:47:37,980 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536057516_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,980 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:37,980 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FF5AA71693964EC38004075268FB4688, taskAttemptId=attempt_1463536057516_0001_1_01_000000_0, creationTime=1463536057902, allocationTime=1463536057905, startTime=1463536057907, finishTime=1463536057980, timeTaken=73, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:37,981 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:37,984 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:37,984 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536057516_0001_1_01_000000_0
2016-05-18 01:47:37,985 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536057516_0001_00_000002
2016-05-18 01:47:37,985 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:TASK_FINISHED]: vertexName=FF5AA71693964EC38004075268FB4688, taskId=task_1463536057516_0001_1_01_000000, startTime=1463536057907, finishTime=1463536057984, timeTaken=77, status=SUCCEEDED, successfulAttemptID=attempt_1463536057516_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536057963, Process_Duration=10, Process_End_Time=1463536057973, Read_Duration=2, Tuples_Read=13, Tuples_Written=23, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=23
2016-05-18 01:47:37,986 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:37,986 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688]
2016-05-18 01:47:37,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:VERTEX_FINISHED]: vertexName=FF5AA71693964EC38004075268FB4688, vertexId=vertex_1463536057516_0001_1_01, initRequestedTime=1463536057714, initedTime=1463536057723, startRequestedTime=1463536057740, startedTime=1463536057741, finishTime=1463536057986, timeTaken=245, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536057963, Process_Duration=10, Process_End_Time=1463536057973, Read_Duration=2, Tuples_Read=13, Tuples_Written=23, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=23, vertexStats=firstTaskStartTime=1463536057907, firstTasksToStart=[ task_1463536057516_0001_1_01_000000 ], lastTaskFinishTime=1463536057980, lastTasksToFinish=[ task_1463536057516_0001_1_01_000000 ], minTaskDuration=73, maxTaskDuration=73, avgTaskDuration=73.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536057516_0001_1_01_000000 ], longestDurationTasks=[ task_1463536057516_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:37,988 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:37,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536057516_0001_00_000002, stoppedTime=1463536057988, exitStatus=0
2016-05-18 01:47:37,988 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:37,988 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:37,988 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536057516_0001_1_00 [C66A159D846D48C0873963CD5ECB2C83]
2016-05-18 01:47:37,989 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536057516_0001_1
2016-05-18 01:47:37,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536057516_0001_1
2016-05-18 01:47:37,990 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: BA7DEDE812D947EA94DE91282F1633C8 for vertex: vertex_1463536057516_0001_1_01 [FF5AA71693964EC38004075268FB4688], outputName: BA7DEDE812D947EA94DE91282F1633C8
2016-05-18 01:47:37,991 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536057516_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:37,991 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536057516_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:38,020 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536057516_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:38,020 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536057516_0001_00_000002: 35 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:38,020 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536057516_0001_00_000002, Exiting
2016-05-18 01:47:38,020 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536057516_0001_00_000002
2016-05-18 01:47:38,020 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536057516_0001_00_000002
2016-05-18 01:47:38,021 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536057516_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:38,021 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536057516_0001_00_000002 completed successfully
2016-05-18 01:47:38,025 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:BA7DEDE812D947EA94DE91282F1633C8 of vertex/vertexGroup:FF5AA71693964EC38004075268FB4688 isVertexGroupOutput:false
2016-05-18 01:47:38,025 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:38,028 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536057516_0001_1, queueSize=0
2016-05-18 01:47:38,029 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536057516_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536057516_0001_1, startTime=1463536057711, finishTime=1463536058025, timeTaken=314, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=560, AM_GC_TIME_MILLIS=22, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=36, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072115843, Process_Duration=12, Process_End_Time=2927072115855, Read_Duration=3, Tuples_Read=26, Tuples_Written=36, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=23
2016-05-18 01:47:38,030 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536057516_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:38,030 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536057516_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:38 Completed Dag: dag_1463536057516_0001_1
2016-05-18 01:47:38,031 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536057516_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:38,031 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536057516_0001_1_post
2016-05-18 01:47:38,031 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,031 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:38,032 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:38,032 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple, with id=dag_1463536057516_0001_1
2016-05-18 01:47:38,032 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330] test/(1/1) ...m/testSimpleBuffer/simple, with id=dag_1463536057516_0001_1
2016-05-18 01:47:38,041 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463536057516_0001
2016-05-18 01:47:38,045 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:38,045 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:38,046 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:38,046 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:38,046 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:38,048 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:38,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:38,049 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:38,050 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:38,050 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:38,051 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:38,052 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:38,053 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:38,056 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:38,060 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/74C41E937B6344ED966B8A26A678C357/B262725A026D4D9D8B8706FAEFAD6330/.tez/application_1463536057516_0001
2016-05-18 01:47:38,061 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple/_temporary
2016-05-18 01:47:38,062 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.569
2016-05-18 01:47:38,151 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-18 01:47:38,190 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:38,191 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:38,259 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.033
2016-05-18 01:47:38,290 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.064
2016-05-18 01:47:38,291 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:38,291 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-18 01:47:38,291 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:38,312 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt
2016-05-18 01:47:38,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt
2016-05-18 01:47:38,339 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-18 01:47:38,463 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] parallel execution of steps is enabled: false
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executing total steps: 1
2016-05-18 01:47:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] allocating management threads: 1
2016-05-18 01:47:38,465 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting step: (1/1) ...testJoinerClosure/cogroup
2016-05-18 01:47:38,465 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [lower*upper] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:38,483 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tez session mode enabled: true
2016-05-18 01:47:38,487 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:38,488 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:38,488 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:38,497 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001 doesn't exist and is created
2016-05-18 01:47:38,517 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:38,517 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:38,518 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd
2016-05-18 01:47:38,534 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536058488_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:38,534 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:38,534 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:38,536 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:38,536 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:38,536 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:38,537 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:38,537 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:38,539 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:38,539 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:38,540 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:38,540 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:38,540 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:38,540 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:38,540 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:38,541 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:38,542 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:38,542 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:38,543 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-log-dir
2016-05-18 01:47:38,543 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-log-dir/history.txt.appattempt_1463536058488_0001_000000, maxErrors=10
2016-05-18 01:47:38,543 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:38,543 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536058488_0001_000000, appSubmitTime=1463536058533, launchTime=1463536058533
2016-05-18 01:47:38,553 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42452
2016-05-18 01:47:38,553 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:38,554 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536058488
2016-05-18 01:47:38,557 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536058488_0001_000000, startTime=1463536058557
2016-05-18 01:47:38,557 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:38,618 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:38,618 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:38,618 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463536058488_0001, dagName=[B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 01:47:38,640 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 01:47:38,640 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:38,641 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536058488_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-log-dir/dag_1463536058488_0001_1.dot
2016-05-18 01:47:38,642 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-log-dir/dag_1463536058488_0001_1-tez-dag.pb.txt
2016-05-18 01:47:38,654 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536058488_0001_1
2016-05-18 01:47:38,654 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,655 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 01:47:38 Running Dag: dag_1463536058488_0001_1
2016-05-18 01:47:38,671 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536058488_0001_1, submitTime=1463536058640
2016-05-18 01:47:38,672 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001/tez-conf.pb" } size: 40937 timestamp: 1463536058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:38,672 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,673 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:38,673 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001/tez-conf.pb" } size: 40937 timestamp: 1463536058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:38,674 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,674 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:38,675 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001/tez-conf.pb" } size: 40937 timestamp: 1463536058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:38,675 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,675 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:38,676 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:38,677 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536058488_0001_1, initTime=1463536058676
2016-05-18 01:47:38,677 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536058488_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:38,678 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:38,678 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:DAG_STARTED]: dagID=dag_1463536058488_0001_1, startTime=1463536058678
2016-05-18 01:47:38,678 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463536058488_0001, dagName=[B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-18 01:47:38,678 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536058488_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:38,679 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A499EAD0559841578096B4FDC021E936 : {5A588F2018FE4424B792EE482BD778F1={InputName=5A588F2018FE4424B792EE482BD778F1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:38,679 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 5A588F2018FE4424B792EE482BD778F1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:38,679 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,679 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,679 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,680 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,682 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:38,682 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F6DE773A366449ABB13B10A73C955B74 : {DDE3CD00AFC54C1A9E64345B038D91A7={InputName=DDE3CD00AFC54C1A9E64345B038D91A7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:38,682 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: DDE3CD00AFC54C1A9E64345B038D91A7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:38,682 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,682 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,683 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,682 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 5A588F2018FE4424B792EE482BD778F1 on vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,683 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,684 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:38,684 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,684 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: DDE3CD00AFC54C1A9E64345B038D91A7 on vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,688 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input DDE3CD00AFC54C1A9E64345B038D91A7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:38,690 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:38,690 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,691 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,690 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 5A588F2018FE4424B792EE482BD778F1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:38,691 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536058488_0001_1_02, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=2, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:38,695 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0], numAdditionalOutputs=1
2016-05-18 01:47:38,695 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=DD8DF8D886FB4240B5F7E975F121D1BB, vertex=vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:38,696 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:38,696 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 19 Min splitLength: 52428800 New desired splits: 1 Total length: 19 Original splits: 3
2016-05-18 01:47:38,696 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 19 numLocations: 1 numSplitsPerLocation: 3 numSplitsInGroup: 3 totalLength: 19 numOriginalSplits: 3 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:38,696 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 3
2016-05-18 01:47:38,697 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 5A588F2018FE4424B792EE482BD778F1 on vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,694 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:38,698 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 19 Min splitLength: 52428800 New desired splits: 1 Total length: 19 Original splits: 3
2016-05-18 01:47:38,698 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CF195BA12E3C4619BB166D2E13F6A4E0:DD8DF8D886FB4240B5F7E975F121D1BB using oldmapred API
2016-05-18 01:47:38,698 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=DD8DF8D886FB4240B5F7E975F121D1BB, vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:38,698 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 19 numLocations: 1 numSplitsPerLocation: 3 numSplitsInGroup: 3 totalLength: 19 numOriginalSplits: 3 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:38,699 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 3
2016-05-18 01:47:38,699 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: DDE3CD00AFC54C1A9E64345B038D91A7 on vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, vertexId=vertex_1463536058488_0001_1_02, initRequestedTime=1463536058684, initedTime=1463536058695, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:38,700 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:38,703 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:38,704 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {5A588F2018FE4424B792EE482BD778F1=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:38,704 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:38,704 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] parallelism set to 1
2016-05-18 01:47:38,705 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:38,704 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {DDE3CD00AFC54C1A9E64345B038D91A7=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:38,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A499EAD0559841578096B4FDC021E936, vertexId=vertex_1463536058488_0001_1_01, initRequestedTime=1463536058679, initedTime=1463536058706, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:38,706 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] with distanceFromRoot: 0
2016-05-18 01:47:38,706 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:38,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536058488_0001_1_01, startRequestedTime=1463536058684, startedTime=1463536058707
2016-05-18 01:47:38,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536058488_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:38,708 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:38,708 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536058488_0001_1_01 for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] numStartedSources: 1 numSources: 2
2016-05-18 01:47:38,709 INFO impl.VertexImpl (VertexImpl.java:transition(2937)) - Cannot start vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] numStartedSources: 1 numSources: 2
2016-05-18 01:47:38,706 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] parallelism set to 1
2016-05-18 01:47:38,709 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:38,709 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A499EAD0559841578096B4FDC021E936
2016-05-18 01:47:38,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F6DE773A366449ABB13B10A73C955B74, vertexId=vertex_1463536058488_0001_1_00, initRequestedTime=1463536058682, initedTime=1463536058710, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:38,710 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] with distanceFromRoot: 0
2016-05-18 01:47:38,710 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,711 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:38,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536058488_0001_1_00, startRequestedTime=1463536058684, startedTime=1463536058711
2016-05-18 01:47:38,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536058488_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:38,712 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] submitted tez dag to app master: application_1463536058488_0001, with dag id: dag_1463536058488_0001_1
2016-05-18 01:47:38,713 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:38,715 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F6DE773A366449ABB13B10A73C955B74
2016-05-18 01:47:38,715 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,715 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_STARTED]: vertexName=A499EAD0559841578096B4FDC021E936, taskId=task_1463536058488_0001_1_01_000000, scheduledTime=1463536058715, launchTime=1463536058715
2016-05-18 01:47:38,716 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536058488_0001_1_00 for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] numStartedSources: 2 numSources: 2
2016-05-18 01:47:38,716 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] with distanceFromRoot: 1
2016-05-18 01:47:38,716 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_STARTED]: vertexName=F6DE773A366449ABB13B10A73C955B74, taskId=task_1463536058488_0001_1_00_000000, scheduledTime=1463536058716, launchTime=1463536058716
2016-05-18 01:47:38,718 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536058488_0001_1_02, startRequestedTime=1463536058716, startedTime=1463536058719
2016-05-18 01:47:38,719 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:38,720 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,722 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:38,724 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CF195BA12E3C4619BB166D2E13F6A4E0 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:38,725 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A499EAD0559841578096B4FDC021E936 in vertex: CF195BA12E3C4619BB166D2E13F6A4E0 numBipartiteSourceTasks: 1
2016-05-18 01:47:38,725 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F6DE773A366449ABB13B10A73C955B74 in vertex: CF195BA12E3C4619BB166D2E13F6A4E0 numBipartiteSourceTasks: 2
2016-05-18 01:47:38,729 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536058488_0001_00_000002, launchTime=1463536058729
2016-05-18 01:47:38,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536058488_0001_00_000001, launchTime=1463536058728
2016-05-18 01:47:38,731 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000001
2016-05-18 01:47:38,731 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000002
2016-05-18 01:47:38,733 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536058488_0001_1_00_000000_0] started. Is using containerId: [container_1463536058488_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:38,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F6DE773A366449ABB13B10A73C955B74, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0, startTime=1463536058733, containerId=container_1463536058488_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:38,734 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536058488_0001_1_01_000000_0] started. Is using containerId: [container_1463536058488_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:38,734 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A499EAD0559841578096B4FDC021E936, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0, startTime=1463536058734, containerId=container_1463536058488_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:38 Starting to run new task attempt: attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,735 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,735 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,735 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:38,736 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: F6DE773A366449ABB13B10A73C955B74, VertexParallelism: 1, TaskAttemptID:attempt_1463536058488_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DDE3CD00AFC54C1A9E64345B038D91A7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF195BA12E3C4619BB166D2E13F6A4E0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:38,736 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:38,736 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,737 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,739 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536058488_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:38,740 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536058488_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:38 Starting to run new task attempt: attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,740 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,740 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,741 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:38,741 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: A499EAD0559841578096B4FDC021E936, VertexParallelism: 1, TaskAttemptID:attempt_1463536058488_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=5A588F2018FE4424B792EE482BD778F1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF195BA12E3C4619BB166D2E13F6A4E0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:38,742 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:38,743 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,744 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536058488_0001_1_01_000000_0 sent events: (0-1).
2016-05-18 01:47:38,745 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536058488_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:38,765 INFO input.MRInput (MRInput.java:initialize(440)) - DDE3CD00AFC54C1A9E64345B038D91A7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:38,765 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: DDE3CD00AFC54C1A9E64345B038D91A7
2016-05-18 01:47:38,766 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:38,767 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:38,770 INFO input.MRInput (MRInput.java:initialize(440)) - 5A588F2018FE4424B792EE482BD778F1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:38,771 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 5A588F2018FE4424B792EE482BD778F1
2016-05-18 01:47:38,777 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A499EAD0559841578096B4FDC021E936, ordinal: 1
2016-05-18 01:47:38,777 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A499EAD0559841578096B4FDC021E936, mem on start (mb), free: 110, total: 484, max: 672
2016-05-18 01:47:38,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:38,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:38,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:38,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:38,777 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[5A588F2018FE4424B792EE482BD778F1:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CF195BA12E3C4619BB166D2E13F6A4E0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:38,778 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:38,778 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F6DE773A366449ABB13B10A73C955B74, ordinal: 0
2016-05-18 01:47:38,778 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:38,778 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:38,778 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[5A588F2018FE4424B792EE482BD778F1:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CF195BA12E3C4619BB166D2E13F6A4E0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:38,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:38,778 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F6DE773A366449ABB13B10A73C955B74, mem on start (mb), free: 110, total: 484, max: 672
2016-05-18 01:47:38,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:38,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:38,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:38,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:38,779 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DDE3CD00AFC54C1A9E64345B038D91A7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CF195BA12E3C4619BB166D2E13F6A4E0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:38,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:38,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:38,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:38,780 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DDE3CD00AFC54C1A9E64345B038D91A7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CF195BA12E3C4619BB166D2E13F6A4E0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:38,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:38,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DDE3CD00AFC54C1A9E64345B038D91A7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:38,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:38,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:38,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 5A588F2018FE4424B792EE482BD778F1
2016-05-18 01:47:38,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 5A588F2018FE4424B792EE482BD778F1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:38,781 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:38,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:38,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:38,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DDE3CD00AFC54C1A9E64345B038D91A7
2016-05-18 01:47:38,783 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:38,786 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,788 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 01:47:38,791 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt"] streamed: true, id: 5A588F2018FE4424B792EE482BD778F1
2016-05-18 01:47:38,791 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: AB034F9E963C4BBEABC21347C36359DB
2016-05-18 01:47:38,792 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 01:47:38,793 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt"] streamed: true, id: DDE3CD00AFC54C1A9E64345B038D91A7
2016-05-18 01:47:38,793 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: AB034F9E963C4BBEABC21347C36359DB
2016-05-18 01:47:38,793 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] AB034F9E963C4BBEABC21347C36359DB
2016-05-18 01:47:38,800 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] AB034F9E963C4BBEABC21347C36359DB
2016-05-18 01:47:38,803 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CF195BA12E3C4619BB166D2E13F6A4E0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@7451dd87, comparator=cascading.tuple.hadoop.util.TupleComparator@2aa705ba, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:38,807 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CF195BA12E3C4619BB166D2E13F6A4E0: , UsingHashComparator=false
2016-05-18 01:47:38,806 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt:0+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt:9+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt:18+1], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=19}
2016-05-18 01:47:38,829 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 5A588F2018FE4424B792EE482BD778F1 initialized RecordReader from event
2016-05-18 01:47:38,805 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt:0+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt:9+9, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt:18+1], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=19}
2016-05-18 01:47:38,825 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CF195BA12E3C4619BB166D2E13F6A4E0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1f0c039e, comparator=cascading.tuple.hadoop.util.TupleComparator@4dc189fe, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:38,838 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CF195BA12E3C4619BB166D2E13F6A4E0: , UsingHashComparator=false
2016-05-18 01:47:38,838 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - DDE3CD00AFC54C1A9E64345B038D91A7 initialized RecordReader from event
2016-05-18 01:47:38,838 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:38,853 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:38,853 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CF195BA12E3C4619BB166D2E13F6A4E0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:38,854 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CF195BA12E3C4619BB166D2E13F6A4E0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:38,851 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:38,855 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:38,855 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CF195BA12E3C4619BB166D2E13F6A4E0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:38,855 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CF195BA12E3C4619BB166D2E13F6A4E0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:38,855 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower.txt"]
2016-05-18 01:47:38,856 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A499EAD0559841578096B4FDC021E936, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:38,856 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/upper.txt"]
2016-05-18 01:47:38,857 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F6DE773A366449ABB13B10A73C955B74, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:38,860 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,860 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F6DE773A366449ABB13B10A73C955B74, mem on close (mb), free: 54, total: 489, max: 672
2016-05-18 01:47:38,860 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,861 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A499EAD0559841578096B4FDC021E936, mem on close (mb), free: 54, total: 489, max: 672
2016-05-18 01:47:38,861 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CF195BA12E3C4619BB166D2E13F6A4E0: Starting flush of map output
2016-05-18 01:47:38,861 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CF195BA12E3C4619BB166D2E13F6A4E0: Starting flush of map output
2016-05-18 01:47:38,862 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CF195BA12E3C4619BB166D2E13F6A4E0: Span0.length = 5, perItem = 15
2016-05-18 01:47:38,861 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CF195BA12E3C4619BB166D2E13F6A4E0: Span0.length = 5, perItem = 15
2016-05-18 01:47:38,863 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CF195BA12E3C4619BB166D2E13F6A4E0: done sorting span=0, length=5, time=0
2016-05-18 01:47:38,863 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CF195BA12E3C4619BB166D2E13F6A4E0: done sorting span=0, length=5, time=0
2016-05-18 01:47:38,863 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CF195BA12E3C4619BB166D2E13F6A4E0: Heap = SpanIterator<0:4> (span=Span[16777216,75]),
2016-05-18 01:47:38,864 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CF195BA12E3C4619BB166D2E13F6A4E0: Heap = SpanIterator<0:4> (span=Span[16777216,75]),
2016-05-18 01:47:38,866 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CF195BA12E3C4619BB166D2E13F6A4E0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-local-dir/output/attempt_1463536058488_0001_1_01_000000_0_10023_0/file.out
2016-05-18 01:47:38,867 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CF195BA12E3C4619BB166D2E13F6A4E0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-local-dir/output/attempt_1463536058488_0001_1_00_000000_0_10021_0/file.out
2016-05-18 01:47:38,870 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:38 Completed running task attempt: attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,871 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:38,871 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536058488_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:38,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536058488_0001_1_01_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536058856, Process_Duration=4, Process_End_Time=1463536058860, Read_Duration=4, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 01:47:38,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:38,878 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536058488_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,878 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A499EAD0559841578096B4FDC021E936, taskAttemptId=attempt_1463536058488_0001_1_01_000000_0, creationTime=1463536058716, allocationTime=1463536058726, startTime=1463536058734, finishTime=1463536058878, timeTaken=144, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:38,879 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,879 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:38,879 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,879 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536058488_0001_1_01_000000_0
2016-05-18 01:47:38,880 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_FINISHED]: vertexName=A499EAD0559841578096B4FDC021E936, taskId=task_1463536058488_0001_1_01_000000, startTime=1463536058734, finishTime=1463536058879, timeTaken=145, status=SUCCEEDED, successfulAttemptID=attempt_1463536058488_0001_1_01_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536058856, Process_Duration=4, Process_End_Time=1463536058860, Read_Duration=4, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 01:47:38,881 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:38,881 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38 Completed running task attempt: attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,881 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:38,882 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536058488_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:38,884 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536058488_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536058857, Process_Duration=2, Process_End_Time=1463536058859, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 01:47:38,882 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000001
2016-05-18 01:47:38,885 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_FINISHED]: vertexName=A499EAD0559841578096B4FDC021E936, vertexId=vertex_1463536058488_0001_1_01, initRequestedTime=1463536058679, initedTime=1463536058706, startRequestedTime=1463536058684, startedTime=1463536058707, finishTime=1463536058881, timeTaken=174, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536058856, Process_Duration=4, Process_End_Time=1463536058860, Read_Duration=4, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463536058734, firstTasksToStart=[ task_1463536058488_0001_1_01_000000 ], lastTaskFinishTime=1463536058878, lastTasksToFinish=[ task_1463536058488_0001_1_01_000000 ], minTaskDuration=144, maxTaskDuration=144, avgTaskDuration=144.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536058488_0001_1_01_000000 ], longestDurationTasks=[ task_1463536058488_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:38,885 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536058488_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,886 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:38,886 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] attempt: attempt_1463536058488_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:38,886 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,887 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:38,887 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536058488_0001_00_000001, stoppedTime=1463536058887, exitStatus=0
2016-05-18 01:47:38,887 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000002
2016-05-18 01:47:38,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F6DE773A366449ABB13B10A73C955B74, taskAttemptId=attempt_1463536058488_0001_1_00_000000_0, creationTime=1463536058716, allocationTime=1463536058727, startTime=1463536058733, finishTime=1463536058887, timeTaken=154, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:38,888 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:38,888 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:38,888 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536058488_0001_1_00_000000_0
2016-05-18 01:47:38,889 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536058488_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:38,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_FINISHED]: vertexName=F6DE773A366449ABB13B10A73C955B74, taskId=task_1463536058488_0001_1_00_000000, startTime=1463536058733, finishTime=1463536058889, timeTaken=156, status=SUCCEEDED, successfulAttemptID=attempt_1463536058488_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536058857, Process_Duration=2, Process_End_Time=1463536058859, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 01:47:38,891 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:38,891 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,891 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536058488_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:38,891 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:38,891 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536058488_0001_00_000001, Exiting
2016-05-18 01:47:38,892 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536058488_0001_00_000001
2016-05-18 01:47:38,892 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536058488_0001_00_000001
2016-05-18 01:47:38,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_FINISHED]: vertexName=F6DE773A366449ABB13B10A73C955B74, vertexId=vertex_1463536058488_0001_1_00, initRequestedTime=1463536058682, initedTime=1463536058710, startRequestedTime=1463536058684, startedTime=1463536058711, finishTime=1463536058891, timeTaken=180, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536058857, Process_Duration=2, Process_End_Time=1463536058859, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463536058733, firstTasksToStart=[ task_1463536058488_0001_1_00_000000 ], lastTaskFinishTime=1463536058887, lastTasksToFinish=[ task_1463536058488_0001_1_00_000000 ], minTaskDuration=154, maxTaskDuration=154, avgTaskDuration=154.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536058488_0001_1_00_000000 ], longestDurationTasks=[ task_1463536058488_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:38,893 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:38,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536058488_0001_00_000002, stoppedTime=1463536058893, exitStatus=0
2016-05-18 01:47:38,894 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] attempt: attempt_1463536058488_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:38,896 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:38,896 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:38,897 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536058488_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:38,897 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536058488_0001_00_000001 completed successfully
2016-05-18 01:47:38,896 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CF195BA12E3C4619BB166D2E13F6A4E0
2016-05-18 01:47:38,897 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536058488_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:38,898 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A499EAD0559841578096B4FDC021E936, destinationVertexName=CF195BA12E3C4619BB166D2E13F6A4E0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:38,898 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F6DE773A366449ABB13B10A73C955B74, destinationVertexName=CF195BA12E3C4619BB166D2E13F6A4E0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:38,898 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,898 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536058488_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:38,898 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:38,898 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536058488_0001_00_000002, Exiting
2016-05-18 01:47:38,898 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536058488_0001_00_000002
2016-05-18 01:47:38,898 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536058488_0001_00_000002
2016-05-18 01:47:38,899 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_STARTED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, taskId=task_1463536058488_0001_1_02_000000, scheduledTime=1463536058898, launchTime=1463536058898
2016-05-18 01:47:38,899 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,899 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536058488_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:38,900 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536058488_0001_00_000002 completed successfully
2016-05-18 01:47:38,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536058488_0001_00_000003, launchTime=1463536058901
2016-05-18 01:47:38,902 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000003
2016-05-18 01:47:38,903 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536058488_0001_1_02_000000_0] started. Is using containerId: [container_1463536058488_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-18 01:47:38,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0, startTime=1463536058903, containerId=container_1463536058488_0001_00_000003, nodeId=127.0.0.1:0
2016-05-18 01:47:38,904 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38 Starting to run new task attempt: attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,905 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,906 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,906 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:38,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: CF195BA12E3C4619BB166D2E13F6A4E0, VertexParallelism: 1, TaskAttemptID:attempt_1463536058488_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A499EAD0559841578096B4FDC021E936, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=F6DE773A366449ABB13B10A73C955B74, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DD8DF8D886FB4240B5F7E975F121D1BB, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:38,907 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=2, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:38,908 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,912 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536058488_0001_1_02_000000_0 2 events [0,2) total 2 vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,912 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536058488_0001_1_02_000000_0, eventCount=2 fromEventId=0 nextFromEventId=2
2016-05-18 01:47:38,916 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:38,917 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:38,916 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:38,927 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CF195BA12E3C4619BB166D2E13F6A4E0, ordinal: 2
2016-05-18 01:47:38,927 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF195BA12E3C4619BB166D2E13F6A4E0, mem on start (mb), free: 47, total: 489, max: 672
2016-05-18 01:47:38,927 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:38,927 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 3 initializers to finish
2016-05-18 01:47:38,927 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:38,927 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:38,930 INFO output.MROutput (MROutput.java:initialize(421)) - DD8DF8D886FB4240B5F7E975F121D1BB: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:38,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:38,931 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F6DE773A366449ABB13B10A73C955B74:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [A499EAD0559841578096B4FDC021E936:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [DD8DF8D886FB4240B5F7E975F121D1BB:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:38,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:38,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.045, finalReserveFractionUsed=0.345
2016-05-18 01:47:38,932 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 3, numScaledRequests: 25, TotalRequested: 422785824, TotalRequestedScaled: 2.0293719551999998E8, TotalJVMHeap: 704643072, TotalAvailable: 461541212, TotalRequested/TotalJVMHeap:0.60
2016-05-18 01:47:38,932 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F6DE773A366449ABB13B10A73C955B74:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [A499EAD0559841578096B4FDC021E936:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [DD8DF8D886FB4240B5F7E975F121D1BB:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:38,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:38,933 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A499EAD0559841578096B4FDC021E936 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:38,935 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A499EAD0559841578096B4FDC021E936: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:38,935 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F6DE773A366449ABB13B10A73C955B74 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:38,936 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A499EAD0559841578096B4FDC021E936: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:38,936 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F6DE773A366449ABB13B10A73C955B74: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:38,936 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F6DE773A366449ABB13B10A73C955B74: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:38,936 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 2
2016-05-18 01:47:38,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 2 IOs to start
2016-05-18 01:47:38,938 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A499EAD0559841578096B4FDC021E936 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:38,938 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F6DE773A366449ABB13B10A73C955B74 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:38,941 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F6DE773A366449ABB13B10A73C955B74
2016-05-18 01:47:38,941 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:38,941 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A499EAD0559841578096B4FDC021E936
2016-05-18 01:47:38,942 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:38,942 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,945 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A499EAD0559841578096B4FDC021E936: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:38,945 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F6DE773A366449ABB13B10A73C955B74: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:38,949 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536058488_0001_1_01_000000_0_10023} to DISK_DIRECT, csize=95, dsize=91, EndTime=1463536058949, TimeTaken=1, Rate=0.09 MB/s
2016-05-18 01:47:38,949 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A499EAD0559841578096B4FDC021E936
2016-05-18 01:47:38,950 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:38,950 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A499EAD0559841578096B4FDC021E936, wasInterrupted=false
2016-05-18 01:47:38,950 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:38,951 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']]
2016-05-18 01:47:38,951 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536058488_0001_1_00_000000_0_10021} to DISK_DIRECT, csize=95, dsize=91, EndTime=1463536058951, TimeTaken=2, Rate=0.05 MB/s
2016-05-18 01:47:38,951 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F6DE773A366449ABB13B10A73C955B74
2016-05-18 01:47:38,952 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:38,952 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F6DE773A366449ABB13B10A73C955B74, wasInterrupted=false
2016-05-18 01:47:38,951 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:38,952 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:38,952 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] streamed: true, id: AB034F9E963C4BBEABC21347C36359DB
2016-05-18 01:47:38,951 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A499EAD0559841578096B4FDC021E936: Already shutdown. Ignoring fetch complete
2016-05-18 01:47:38,953 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"], id: DD8DF8D886FB4240B5F7E975F121D1BB
2016-05-18 01:47:38,953 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-local-dir/output/attempt_1463536058488_0001_1_01_000000_0_10023/file.out, len=95, isLocal=true
2016-05-18 01:47:38,954 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 95 bytes from disk
2016-05-18 01:47:38,954 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:38,954 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:38,953 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:38,954 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-18 01:47:38,954 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A499EAD0559841578096B4FDC021E936
2016-05-18 01:47:38,954 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A499EAD0559841578096B4FDC021E936: Shuffle Runner thread complete
2016-05-18 01:47:38,955 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001_wd/localmode-local-dir/output/attempt_1463536058488_0001_1_00_000000_0_10021/file.out, len=95, isLocal=true
2016-05-18 01:47:38,955 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-18 01:47:38,955 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] AB034F9E963C4BBEABC21347C36359DB, for 2 inputs
2016-05-18 01:47:38,955 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] AB034F9E963C4BBEABC21347C36359DB, for 2 inputs
2016-05-18 01:47:38,955 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 95 bytes from disk
2016-05-18 01:47:38,955 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:38,956 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:38,956 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-18 01:47:38,956 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F6DE773A366449ABB13B10A73C955B74
2016-05-18 01:47:38,956 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F6DE773A366449ABB13B10A73C955B74: Shuffle Runner thread complete
2016-05-18 01:47:38,962 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:38,962 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-18 01:47:38,966 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CF195BA12E3C4619BB166D2E13F6A4E0, all 2 inputs ready in: 00:00:00.000
2016-05-18 01:47:38,967 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A499EAD0559841578096B4FDC021E936: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:38,969 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F6DE773A366449ABB13B10A73C955B74: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:38,976 INFO output.MROutput (MROutput.java:close(526)) - DD8DF8D886FB4240B5F7E975F121D1BB closed
2016-05-18 01:47:38,977 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536058488_0001_1_02_000000_0 given a go for committing the task output.
2016-05-18 01:47:38,978 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,978 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF195BA12E3C4619BB166D2E13F6A4E0, mem on close (mb), free: 41, total: 489, max: 672
2016-05-18 01:47:38,978 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A499EAD0559841578096B4FDC021E936
2016-05-18 01:47:38,978 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A499EAD0559841578096B4FDC021E936: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:38,979 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F6DE773A366449ABB13B10A73C955B74
2016-05-18 01:47:38,979 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F6DE773A366449ABB13B10A73C955B74: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:38,979 INFO output.MROutput (MROutput.java:close(526)) - DD8DF8D886FB4240B5F7E975F121D1BB closed
2016-05-18 01:47:38,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:38 Completed running task attempt: attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,980 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0, askedToStop=false
2016-05-18 01:47:38,980 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536058488_0001_1_02_000000_0, stopRequested=false
2016-05-18 01:47:38,980 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536058488_0001_1_02_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=18, LAST_EVENT_RECEIVED=18][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536058966, Process_Duration=12, Process_End_Time=1463536058978, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 01:47:38,983 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536058488_0001_1_02_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,984 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:38,984 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:38,984 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, taskAttemptId=attempt_1463536058488_0001_1_02_000000_0, creationTime=1463536058899, allocationTime=1463536058900, startTime=1463536058903, finishTime=1463536058984, timeTaken=81, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:38,984 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536058488_0001_1_02_000000_0
2016-05-18 01:47:38,985 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:38,985 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536058488_0001_00_000003
2016-05-18 01:47:38,985 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:TASK_FINISHED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, taskId=task_1463536058488_0001_1_02_000000, startTime=1463536058903, finishTime=1463536058984, timeTaken=81, status=SUCCEEDED, successfulAttemptID=attempt_1463536058488_0001_1_02_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=18, LAST_EVENT_RECEIVED=18, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536058966, Process_Duration=12, Process_End_Time=1463536058978, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 01:47:38,986 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:38,986 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0]
2016-05-18 01:47:38,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:VERTEX_FINISHED]: vertexName=CF195BA12E3C4619BB166D2E13F6A4E0, vertexId=vertex_1463536058488_0001_1_02, initRequestedTime=1463536058684, initedTime=1463536058695, startRequestedTime=1463536058716, startedTime=1463536058719, finishTime=1463536058986, timeTaken=267, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=18, LAST_EVENT_RECEIVED=18, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536058966, Process_Duration=12, Process_End_Time=1463536058978, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463536058903, firstTasksToStart=[ task_1463536058488_0001_1_02_000000 ], lastTaskFinishTime=1463536058984, lastTasksToFinish=[ task_1463536058488_0001_1_02_000000 ], minTaskDuration=81, maxTaskDuration=81, avgTaskDuration=81.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536058488_0001_1_02_000000 ], longestDurationTasks=[ task_1463536058488_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:38,988 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:38,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536058488_0001_00_000003, stoppedTime=1463536058988, exitStatus=0
2016-05-18 01:47:38,988 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-18 01:47:38,988 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:38,989 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536058488_0001_1_01 [A499EAD0559841578096B4FDC021E936]
2016-05-18 01:47:38,989 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536058488_0001_1_00 [F6DE773A366449ABB13B10A73C955B74]
2016-05-18 01:47:38,989 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536058488_0001_1
2016-05-18 01:47:38,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536058488_0001_1
2016-05-18 01:47:38,989 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: DD8DF8D886FB4240B5F7E975F121D1BB for vertex: vertex_1463536058488_0001_1_02 [CF195BA12E3C4619BB166D2E13F6A4E0], outputName: DD8DF8D886FB4240B5F7E975F121D1BB
2016-05-18 01:47:38,993 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536058488_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:38,993 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536058488_0001_00_000003. Relying on regular task shutdown for it to end
2016-05-18 01:47:39,007 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536058488_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-18 01:47:39,007 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536058488_0001_00_000003: 22 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:39,007 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536058488_0001_00_000003, Exiting
2016-05-18 01:47:39,007 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536058488_0001_00_000003
2016-05-18 01:47:39,007 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536058488_0001_00_000003
2016-05-18 01:47:39,008 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536058488_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:39,008 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536058488_0001_00_000003 completed successfully
2016-05-18 01:47:39,012 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:DD8DF8D886FB4240B5F7E975F121D1BB of vertex/vertexGroup:CF195BA12E3C4619BB166D2E13F6A4E0 isVertexGroupOutput:false
2016-05-18 01:47:39,012 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3, commitInProgress=0, terminationCause=null
2016-05-18 01:47:39,015 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536058488_0001_1, queueSize=0
2016-05-18 01:47:39,016 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536058488_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536058488_0001_1, startTime=1463536058678, finishTime=1463536059012, timeTaken=334, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=3, TOTAL_LAUNCHED_TASKS=3, RACK_LOCAL_TASKS=2, AM_CPU_MILLISECONDS=760, AM_GC_TIME_MILLIS=28, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=10, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=15, OUTPUT_BYTES=150, OUTPUT_BYTES_WITH_OVERHEAD=182, OUTPUT_BYTES_PHYSICAL=190, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=2, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=31, MERGE_PHASE_TIME=39, FIRST_EVENT_RECEIVED=18, LAST_EVENT_RECEIVED=18, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=4390608176679, Process_Duration=18, Process_End_Time=4390608176697, Read_Duration=7, Tuples_Read=20, Tuples_Written=15, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=10, Tuples_Written=5
2016-05-18 01:47:39,016 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536058488_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:39,016 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536058488_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:39 Completed Dag: dag_1463536058488_0001_1
2016-05-18 01:47:39,017 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536058488_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:39,017 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536058488_0001_1_post
2016-05-18 01:47:39,017 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,017 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:39,017 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:39,018 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup, with id=dag_1463536058488_0001_1
2016-05-18 01:47:39,018 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600] lower*upper/(1/1) ...testJoinerClosure/cogroup, with id=dag_1463536058488_0001_1
2016-05-18 01:47:39,027 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463536058488_0001
2016-05-18 01:47:39,031 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:39,031 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:39,031 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:39,031 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:39,031 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:39,032 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:39,032 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:39,034 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:39,034 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:39,034 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:39,035 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:39,035 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:39,036 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:39,036 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:39,040 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/B1F70EC5B7E44AC19CC638AB0A424183/D67B19ECB52B476DA7B4A8A198B44600/.tez/application_1463536058488_0001
2016-05-18 01:47:39,041 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup/_temporary
2016-05-18 01:47:39,042 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] completed in: 00:00.578
2016-05-18 01:47:39,133 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-18 01:47:39,162 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:39,163 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:39,224 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.025
2016-05-18 01:47:39,243 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.044
2016-05-18 01:47:39,244 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:39,244 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:39,245 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:39,262 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:39,271 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-18 01:47:39,331 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:39,331 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:39,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-18 01:47:39,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:39,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:39,332 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:39,333 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer2/simple2
2016-05-18 01:47:39,333 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:39,349 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:39,353 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:39,354 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:39,354 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:39,364 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001 doesn't exist and is created
2016-05-18 01:47:39,383 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:39,384 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:39,385 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd
2016-05-18 01:47:39,402 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536059354_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:39,402 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:39,402 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:39,404 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:39,404 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:39,405 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:39,405 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:39,406 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:39,407 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:39,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:39,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:39,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:39,408 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:39,408 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:39,409 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:39,409 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:39,409 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:39,409 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:39,410 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:39,410 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:39,412 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:39,413 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:39,413 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:39,414 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-log-dir
2016-05-18 01:47:39,414 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-log-dir/history.txt.appattempt_1463536059354_0001_000000, maxErrors=10
2016-05-18 01:47:39,415 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:39,415 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536059354_0001_000000, appSubmitTime=1463536059401, launchTime=1463536059402
2016-05-18 01:47:39,423 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:39,428 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47392
2016-05-18 01:47:39,428 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536059354
2016-05-18 01:47:39,430 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536059354_0001_000000, startTime=1463536059430
2016-05-18 01:47:39,430 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:39,484 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:39,484 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:39,484 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463536059354_0001, dagName=[3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 01:47:39,504 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 01:47:39,504 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:39,505 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536059354_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-log-dir/dag_1463536059354_0001_1.dot
2016-05-18 01:47:39,506 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-log-dir/dag_1463536059354_0001_1-tez-dag.pb.txt
2016-05-18 01:47:39,521 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536059354_0001_1
2016-05-18 01:47:39,521 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,521 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 01:47:39 Running Dag: dag_1463536059354_0001_1
2016-05-18 01:47:39,538 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536059354_0001_1, submitTime=1463536059504
2016-05-18 01:47:39,539 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001/tez-conf.pb" } size: 40937 timestamp: 1463536059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:39,539 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,539 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:39,540 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001/tez-conf.pb" } size: 40937 timestamp: 1463536059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:39,540 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,540 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:39,540 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:39,541 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536059354_0001_1, initTime=1463536059541
2016-05-18 01:47:39,541 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536059354_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:39,543 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:39,544 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:DAG_STARTED]: dagID=dag_1463536059354_0001_1, startTime=1463536059543
2016-05-18 01:47:39,544 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463536059354_0001, dagName=[3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-18 01:47:39,544 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536059354_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:39,545 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B8EFD2E8DAFA4D18A05AE4340B36F23D : {98578306CD6F40D59EF12AEC531EF9EA={InputName=98578306CD6F40D59EF12AEC531EF9EA}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:39,546 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 98578306CD6F40D59EF12AEC531EF9EA, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:39,546 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,546 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,546 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,546 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,547 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:39,547 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,548 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 98578306CD6F40D59EF12AEC531EF9EA on vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,551 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:39,551 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,551 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,551 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 98578306CD6F40D59EF12AEC531EF9EA asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:39,554 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536059354_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:39,554 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125], numAdditionalOutputs=1
2016-05-18 01:47:39,555 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6D83D38BEBB54D77A26446E072DF6318, vertex=vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:39,557 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:39,557 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:39,557 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:39,558 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:39,558 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F41DF5F52D2D4F5CA073E4C012D71125:6D83D38BEBB54D77A26446E072DF6318 using oldmapred API
2016-05-18 01:47:39,558 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6D83D38BEBB54D77A26446E072DF6318, vertexName=F41DF5F52D2D4F5CA073E4C012D71125, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:39,559 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 98578306CD6F40D59EF12AEC531EF9EA on vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,561 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, vertexId=vertex_1463536059354_0001_1_01, initRequestedTime=1463536059547, initedTime=1463536059554, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:39,561 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:39,562 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:39,562 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {98578306CD6F40D59EF12AEC531EF9EA=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:39,563 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] parallelism set to 1
2016-05-18 01:47:39,563 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:39,564 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, vertexId=vertex_1463536059354_0001_1_00, initRequestedTime=1463536059545, initedTime=1463536059564, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:39,564 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] with distanceFromRoot: 0
2016-05-18 01:47:39,565 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:39,565 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536059354_0001_1_00, startRequestedTime=1463536059547, startedTime=1463536059565
2016-05-18 01:47:39,566 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536059354_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:39,564 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536059354_0001, with dag id: dag_1463536059354_0001_1
2016-05-18 01:47:39,566 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:39,567 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536059354_0001_1_00 for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] numStartedSources: 1 numSources: 1
2016-05-18 01:47:39,567 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B8EFD2E8DAFA4D18A05AE4340B36F23D
2016-05-18 01:47:39,567 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,567 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] with distanceFromRoot: 1
2016-05-18 01:47:39,569 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536059354_0001_1_01, startRequestedTime=1463536059567, startedTime=1463536059569
2016-05-18 01:47:39,570 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:39,571 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F41DF5F52D2D4F5CA073E4C012D71125 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:39,570 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_STARTED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, taskId=task_1463536059354_0001_1_00_000000, scheduledTime=1463536059570, launchTime=1463536059570
2016-05-18 01:47:39,572 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,572 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B8EFD2E8DAFA4D18A05AE4340B36F23D in vertex: F41DF5F52D2D4F5CA073E4C012D71125 numBipartiteSourceTasks: 1
2016-05-18 01:47:39,574 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:39,576 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536059354_0001_00_000001, launchTime=1463536059576
2016-05-18 01:47:39,577 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536059354_0001_00_000001
2016-05-18 01:47:39,577 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536059354_0001_1_00_000000_0] started. Is using containerId: [container_1463536059354_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:39,578 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0, startTime=1463536059577, containerId=container_1463536059354_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:39,578 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536059354_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39 Starting to run new task attempt: attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,579 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,579 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,580 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:39,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: B8EFD2E8DAFA4D18A05AE4340B36F23D, VertexParallelism: 1, TaskAttemptID:attempt_1463536059354_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=98578306CD6F40D59EF12AEC531EF9EA, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F41DF5F52D2D4F5CA073E4C012D71125, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:39,580 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:39,581 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,584 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536059354_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:39,584 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536059354_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:39,586 INFO input.MRInput (MRInput.java:initialize(440)) - 98578306CD6F40D59EF12AEC531EF9EA using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:39,586 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:39,587 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 98578306CD6F40D59EF12AEC531EF9EA
2016-05-18 01:47:39,594 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B8EFD2E8DAFA4D18A05AE4340B36F23D, ordinal: 0
2016-05-18 01:47:39,594 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8EFD2E8DAFA4D18A05AE4340B36F23D, mem on start (mb), free: 151, total: 489, max: 672
2016-05-18 01:47:39,594 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:39,594 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:39,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:39,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:39,595 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[98578306CD6F40D59EF12AEC531EF9EA:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F41DF5F52D2D4F5CA073E4C012D71125:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:39,595 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:39,596 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:39,596 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:39,596 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[98578306CD6F40D59EF12AEC531EF9EA:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F41DF5F52D2D4F5CA073E4C012D71125:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:39,596 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:39,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 98578306CD6F40D59EF12AEC531EF9EA being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:39,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:39,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:39,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 98578306CD6F40D59EF12AEC531EF9EA
2016-05-18 01:47:39,598 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:39,598 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,601 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:39,601 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 98578306CD6F40D59EF12AEC531EF9EA initialized RecordReader from event
2016-05-18 01:47:39,603 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:39,604 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 98578306CD6F40D59EF12AEC531EF9EA
2016-05-18 01:47:39,604 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: D3B0905162B14035BABC858526D7B9FE
2016-05-18 01:47:39,604 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] D3B0905162B14035BABC858526D7B9FE
2016-05-18 01:47:39,605 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F41DF5F52D2D4F5CA073E4C012D71125 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2c0cc628, comparator=cascading.tuple.hadoop.util.TupleComparator@7aa582af, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:39,606 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F41DF5F52D2D4F5CA073E4C012D71125: , UsingHashComparator=false
2016-05-18 01:47:39,620 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:39,620 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:39,620 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F41DF5F52D2D4F5CA073E4C012D71125: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:39,621 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F41DF5F52D2D4F5CA073E4C012D71125: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:39,621 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:39,621 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B8EFD2E8DAFA4D18A05AE4340B36F23D, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:39,624 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,624 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8EFD2E8DAFA4D18A05AE4340B36F23D, mem on close (mb), free: 49, total: 489, max: 672
2016-05-18 01:47:39,625 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F41DF5F52D2D4F5CA073E4C012D71125: Starting flush of map output
2016-05-18 01:47:39,625 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F41DF5F52D2D4F5CA073E4C012D71125: Span0.length = 13, perItem = 15
2016-05-18 01:47:39,625 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F41DF5F52D2D4F5CA073E4C012D71125: done sorting span=0, length=13, time=0
2016-05-18 01:47:39,626 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F41DF5F52D2D4F5CA073E4C012D71125: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 01:47:39,628 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F41DF5F52D2D4F5CA073E4C012D71125: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-local-dir/output/attempt_1463536059354_0001_1_00_000000_0_10031_0/file.out
2016-05-18 01:47:39,631 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:39 Completed running task attempt: attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,632 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:39,632 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536059354_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:39,633 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536059354_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536059622, Process_Duration=2, Process_End_Time=1463536059624, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:39,634 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536059354_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:39,634 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:39,634 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:39,634 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,634 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, taskAttemptId=attempt_1463536059354_0001_1_00_000000_0, creationTime=1463536059572, allocationTime=1463536059575, startTime=1463536059577, finishTime=1463536059634, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:39,635 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536059354_0001_1_00_000000_0
2016-05-18 01:47:39,634 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536059354_0001_00_000001
2016-05-18 01:47:39,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_FINISHED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, taskId=task_1463536059354_0001_1_00_000000, startTime=1463536059577, finishTime=1463536059635, timeTaken=58, status=SUCCEEDED, successfulAttemptID=attempt_1463536059354_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536059622, Process_Duration=2, Process_End_Time=1463536059624, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:39,637 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:39,637 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,638 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_FINISHED]: vertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, vertexId=vertex_1463536059354_0001_1_00, initRequestedTime=1463536059545, initedTime=1463536059564, startRequestedTime=1463536059547, startedTime=1463536059565, finishTime=1463536059637, timeTaken=72, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536059622, Process_Duration=2, Process_End_Time=1463536059624, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536059577, firstTasksToStart=[ task_1463536059354_0001_1_00_000000 ], lastTaskFinishTime=1463536059634, lastTasksToFinish=[ task_1463536059354_0001_1_00_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536059354_0001_1_00_000000 ], longestDurationTasks=[ task_1463536059354_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:39,638 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:39,638 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536059354_0001_00_000001, stoppedTime=1463536059638, exitStatus=0
2016-05-18 01:47:39,638 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] attempt: attempt_1463536059354_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:39,638 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:39,638 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F41DF5F52D2D4F5CA073E4C012D71125
2016-05-18 01:47:39,639 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, destinationVertexName=F41DF5F52D2D4F5CA073E4C012D71125. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:39,639 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,638 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:39,639 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536059354_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:39,642 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_STARTED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, taskId=task_1463536059354_0001_1_01_000000, scheduledTime=1463536059640, launchTime=1463536059640
2016-05-18 01:47:39,642 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536059354_0001_00_000002, launchTime=1463536059645
2016-05-18 01:47:39,646 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536059354_0001_00_000002
2016-05-18 01:47:39,647 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536059354_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:39,647 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536059354_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:39,647 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536059354_0001_00_000001, Exiting
2016-05-18 01:47:39,648 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536059354_0001_00_000001
2016-05-18 01:47:39,648 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536059354_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,648 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536059354_0001_1_01_000000_0] started. Is using containerId: [container_1463536059354_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:39,648 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536059354_0001_00_000001
2016-05-18 01:47:39,648 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0, startTime=1463536059648, containerId=container_1463536059354_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:39,649 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536059354_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:39,649 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536059354_0001_00_000001 completed successfully
2016-05-18 01:47:39 Starting to run new task attempt: attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,650 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,650 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,650 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:39,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: F41DF5F52D2D4F5CA073E4C012D71125, VertexParallelism: 1, TaskAttemptID:attempt_1463536059354_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B8EFD2E8DAFA4D18A05AE4340B36F23D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6D83D38BEBB54D77A26446E072DF6318, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:39,651 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:39,652 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,655 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:39,657 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536059354_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,672 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536059354_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:39,673 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:39,684 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F41DF5F52D2D4F5CA073E4C012D71125, ordinal: 1
2016-05-18 01:47:39,684 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F41DF5F52D2D4F5CA073E4C012D71125, mem on start (mb), free: 256, total: 490, max: 672
2016-05-18 01:47:39,684 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:39,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:39,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:39,687 INFO output.MROutput (MROutput.java:initialize(421)) - 6D83D38BEBB54D77A26446E072DF6318: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:39,688 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:39,688 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6D83D38BEBB54D77A26446E072DF6318:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B8EFD2E8DAFA4D18A05AE4340B36F23D:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:39,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:39,689 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:39,689 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:39,689 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6D83D38BEBB54D77A26446E072DF6318:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B8EFD2E8DAFA4D18A05AE4340B36F23D:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:39,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:39,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B8EFD2E8DAFA4D18A05AE4340B36F23D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:39,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:39,690 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:39,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:39,691 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:39,692 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B8EFD2E8DAFA4D18A05AE4340B36F23D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:39,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B8EFD2E8DAFA4D18A05AE4340B36F23D
2016-05-18 01:47:39,695 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:39,695 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,697 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:39,699 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:39,699 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536059354_0001_1_00_000000_0_10031} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463536059699, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 01:47:39,699 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B8EFD2E8DAFA4D18A05AE4340B36F23D
2016-05-18 01:47:39,699 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: D3B0905162B14035BABC858526D7B9FE
2016-05-18 01:47:39,700 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:39,700 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"], id: 6D83D38BEBB54D77A26446E072DF6318
2016-05-18 01:47:39,700 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B8EFD2E8DAFA4D18A05AE4340B36F23D, wasInterrupted=false
2016-05-18 01:47:39,700 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-18 01:47:39,703 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:39,704 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:39,704 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] D3B0905162B14035BABC858526D7B9FE, for 1 inputs
2016-05-18 01:47:39,705 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001_wd/localmode-local-dir/output/attempt_1463536059354_0001_1_00_000000_0_10031/file.out, len=175, isLocal=true
2016-05-18 01:47:39,705 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 01:47:39,705 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:39,705 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:39,705 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 01:47:39,705 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B8EFD2E8DAFA4D18A05AE4340B36F23D
2016-05-18 01:47:39,706 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: Shuffle Runner thread complete
2016-05-18 01:47:39,706 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F41DF5F52D2D4F5CA073E4C012D71125, all 1 inputs ready in: 00:00:00.002
2016-05-18 01:47:39,707 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:39,710 INFO output.MROutput (MROutput.java:close(526)) - 6D83D38BEBB54D77A26446E072DF6318 closed
2016-05-18 01:47:39,711 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536059354_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:39,711 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,712 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F41DF5F52D2D4F5CA073E4C012D71125, mem on close (mb), free: 253, total: 490, max: 672
2016-05-18 01:47:39,712 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B8EFD2E8DAFA4D18A05AE4340B36F23D
2016-05-18 01:47:39,712 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8EFD2E8DAFA4D18A05AE4340B36F23D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:39,712 INFO output.MROutput (MROutput.java:close(526)) - 6D83D38BEBB54D77A26446E072DF6318 closed
2016-05-18 01:47:39,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:39 Completed running task attempt: attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,713 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:39,714 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536059354_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:39,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536059354_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536059706, Process_Duration=5, Process_End_Time=1463536059711, Read_Duration=2, Tuples_Read=13, Tuples_Written=18, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=18]]
2016-05-18 01:47:39,716 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536059354_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:39,717 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:39,717 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:39,717 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,719 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536059354_0001_00_000002
2016-05-18 01:47:39,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, taskAttemptId=attempt_1463536059354_0001_1_01_000000_0, creationTime=1463536059642, allocationTime=1463536059644, startTime=1463536059648, finishTime=1463536059719, timeTaken=71, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:39,719 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536059354_0001_1_01_000000_0
2016-05-18 01:47:39,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:TASK_FINISHED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, taskId=task_1463536059354_0001_1_01_000000, startTime=1463536059648, finishTime=1463536059720, timeTaken=72, status=SUCCEEDED, successfulAttemptID=attempt_1463536059354_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536059706, Process_Duration=5, Process_End_Time=1463536059711, Read_Duration=2, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18
2016-05-18 01:47:39,721 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:39,721 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125]
2016-05-18 01:47:39,722 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:VERTEX_FINISHED]: vertexName=F41DF5F52D2D4F5CA073E4C012D71125, vertexId=vertex_1463536059354_0001_1_01, initRequestedTime=1463536059547, initedTime=1463536059554, startRequestedTime=1463536059567, startedTime=1463536059569, finishTime=1463536059721, timeTaken=152, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536059706, Process_Duration=5, Process_End_Time=1463536059711, Read_Duration=2, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18, vertexStats=firstTaskStartTime=1463536059648, firstTasksToStart=[ task_1463536059354_0001_1_01_000000 ], lastTaskFinishTime=1463536059719, lastTasksToFinish=[ task_1463536059354_0001_1_01_000000 ], minTaskDuration=71, maxTaskDuration=71, avgTaskDuration=71.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536059354_0001_1_01_000000 ], longestDurationTasks=[ task_1463536059354_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:39,723 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:39,723 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536059354_0001_00_000002, stoppedTime=1463536059723, exitStatus=0
2016-05-18 01:47:39,723 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:39,723 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:39,723 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536059354_0001_1_00 [B8EFD2E8DAFA4D18A05AE4340B36F23D]
2016-05-18 01:47:39,724 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536059354_0001_1
2016-05-18 01:47:39,724 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536059354_0001_1
2016-05-18 01:47:39,724 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536059354_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:39,724 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6D83D38BEBB54D77A26446E072DF6318 for vertex: vertex_1463536059354_0001_1_01 [F41DF5F52D2D4F5CA073E4C012D71125], outputName: 6D83D38BEBB54D77A26446E072DF6318
2016-05-18 01:47:39,724 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536059354_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:39,730 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536059354_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:39,730 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536059354_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:39,730 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536059354_0001_00_000002, Exiting
2016-05-18 01:47:39,730 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536059354_0001_00_000002
2016-05-18 01:47:39,730 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536059354_0001_00_000002
2016-05-18 01:47:39,731 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536059354_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:39,731 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536059354_0001_00_000002 completed successfully
2016-05-18 01:47:39,736 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6D83D38BEBB54D77A26446E072DF6318 of vertex/vertexGroup:F41DF5F52D2D4F5CA073E4C012D71125 isVertexGroupOutput:false
2016-05-18 01:47:39,736 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:39,739 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536059354_0001_1, queueSize=0
2016-05-18 01:47:39,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536059354_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536059354_0001_1, startTime=1463536059543, finishTime=1463536059736, timeTaken=193, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=380, AM_GC_TIME_MILLIS=13, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=31, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072119328, Process_Duration=7, Process_End_Time=2927072119335, Read_Duration=3, Tuples_Read=26, Tuples_Written=31, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=18
2016-05-18 01:47:39,739 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536059354_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:39,739 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536059354_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:39 Completed Dag: dag_1463536059354_0001_1
2016-05-18 01:47:39,740 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536059354_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:39,740 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536059354_0001_1_post
2016-05-18 01:47:39,740 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:39,740 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:39,740 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:39,741 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2, with id=dag_1463536059354_0001_1
2016-05-18 01:47:39,741 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9] test/(1/1) ...testSimpleBuffer2/simple2, with id=dag_1463536059354_0001_1
2016-05-18 01:47:39,747 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463536059354_0001
2016-05-18 01:47:39,750 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:39,750 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:39,750 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:39,750 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:39,750 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:39,751 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:39,751 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:39,751 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:39,752 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:39,752 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:39,752 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:39,753 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:39,753 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:39,753 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:39,757 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/3627903EA0BC41D38E84B84DFC1396F6/DB233346F1CE44678E93D16533E65FE9/.tez/application_1463536059354_0001
2016-05-18 01:47:39,757 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2/_temporary
2016-05-18 01:47:39,758 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.426
2016-05-18 01:47:39,836 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-18 01:47:39,865 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:39,866 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:39,910 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 01:47:39,933 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.036
2016-05-18 01:47:39,933 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:39,934 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:39,934 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:39,953 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt
2016-05-18 01:47:39,962 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-18 01:47:40,022 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:40,023 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 01:47:40,023 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-18 01:47:40,024 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:40,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:40,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:40,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer3/simple3
2016-05-18 01:47:40,026 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:40,048 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:40,052 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:40,053 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:40,053 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:40,062 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001 doesn't exist and is created
2016-05-18 01:47:40,082 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:40,082 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:40,083 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd
2016-05-18 01:47:40,098 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536060053_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:40,098 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:40,098 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:40,100 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:40,100 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:40,100 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:40,101 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:40,101 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:40,103 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:40,104 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:40,104 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:40,104 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:40,104 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:40,104 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:40,105 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:40,105 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:40,105 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:40,105 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:40,106 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:40,106 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:40,108 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:40,109 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:40,109 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:40,109 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-log-dir
2016-05-18 01:47:40,110 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-log-dir/history.txt.appattempt_1463536060053_0001_000000, maxErrors=10
2016-05-18 01:47:40,110 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:40,111 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536060053_0001_000000, appSubmitTime=1463536060097, launchTime=1463536060097
2016-05-18 01:47:40,119 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:40,121 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:57388
2016-05-18 01:47:40,122 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536060053
2016-05-18 01:47:40,124 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536060053_0001_000000, startTime=1463536060123
2016-05-18 01:47:40,124 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:40,182 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:40,182 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:40,183 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463536060053_0001, dagName=[FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 01:47:40,202 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 01:47:40,202 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:40,203 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536060053_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-log-dir/dag_1463536060053_0001_1.dot
2016-05-18 01:47:40,204 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-log-dir/dag_1463536060053_0001_1-tez-dag.pb.txt
2016-05-18 01:47:40,211 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536060053_0001_1
2016-05-18 01:47:40,211 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,211 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 01:47:40 Running Dag: dag_1463536060053_0001_1
2016-05-18 01:47:40,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536060053_0001_1, submitTime=1463536060202
2016-05-18 01:47:40,227 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001/tez-conf.pb" } size: 40937 timestamp: 1463536060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:40,228 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,228 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:40,228 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001/tez-conf.pb" } size: 40937 timestamp: 1463536060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:40,229 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,229 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:40,229 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:40,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536060053_0001_1, initTime=1463536060229
2016-05-18 01:47:40,230 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060053_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:40,231 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:40,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:DAG_STARTED]: dagID=dag_1463536060053_0001_1, startTime=1463536060231
2016-05-18 01:47:40,231 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060053_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:40,231 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BF07C4FF717249E9A9967A64882DA52E : {2077A6B81DF44128B1AE45119C3B251F={InputName=2077A6B81DF44128B1AE45119C3B251F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:40,231 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 2077A6B81DF44128B1AE45119C3B251F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:40,231 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,232 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,232 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,232 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,233 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:40,233 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 2077A6B81DF44128B1AE45119C3B251F on vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,233 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,235 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463536060053_0001, dagName=[FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-18 01:47:40,238 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:40,239 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,239 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,241 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 2077A6B81DF44128B1AE45119C3B251F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:40,241 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536060053_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:40,242 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488], numAdditionalOutputs=1
2016-05-18 01:47:40,242 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6C371302A3E04EF8A0C3434252B2F1C4, vertex=vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:40,246 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:40,246 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 30 Min splitLength: 52428800 New desired splits: 1 Total length: 30 Original splits: 2
2016-05-18 01:47:40,246 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 30 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 30 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:40,247 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:40,247 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E1CEA054BD9047869ADC1C7903BDC488:6C371302A3E04EF8A0C3434252B2F1C4 using oldmapred API
2016-05-18 01:47:40,247 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 2077A6B81DF44128B1AE45119C3B251F on vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,249 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6C371302A3E04EF8A0C3434252B2F1C4, vertexName=E1CEA054BD9047869ADC1C7903BDC488, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:40,250 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, vertexId=vertex_1463536060053_0001_1_01, initRequestedTime=1463536060233, initedTime=1463536060242, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:40,250 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:40,251 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:40,252 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {2077A6B81DF44128B1AE45119C3B251F=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:40,252 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] parallelism set to 1
2016-05-18 01:47:40,252 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:40,252 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, vertexId=vertex_1463536060053_0001_1_00, initRequestedTime=1463536060231, initedTime=1463536060252, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:40,253 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] with distanceFromRoot: 0
2016-05-18 01:47:40,253 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:40,253 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536060053_0001_1_00, startRequestedTime=1463536060233, startedTime=1463536060253
2016-05-18 01:47:40,253 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536060053_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:40,253 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:40,253 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536060053_0001_1_00 for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] numStartedSources: 1 numSources: 1
2016-05-18 01:47:40,253 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] with distanceFromRoot: 1
2016-05-18 01:47:40,254 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536060053_0001_1_01, startRequestedTime=1463536060253, startedTime=1463536060253
2016-05-18 01:47:40,255 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:40,256 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E1CEA054BD9047869ADC1C7903BDC488 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:40,256 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BF07C4FF717249E9A9967A64882DA52E
2016-05-18 01:47:40,257 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,258 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_STARTED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, taskId=task_1463536060053_0001_1_00_000000, scheduledTime=1463536060258, launchTime=1463536060258
2016-05-18 01:47:40,258 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,258 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BF07C4FF717249E9A9967A64882DA52E in vertex: E1CEA054BD9047869ADC1C7903BDC488 numBipartiteSourceTasks: 1
2016-05-18 01:47:40,259 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:40,261 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536060053_0001_00_000001, launchTime=1463536060261
2016-05-18 01:47:40,262 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060053_0001_00_000001
2016-05-18 01:47:40,263 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536060053_0001_1_00_000000_0] started. Is using containerId: [container_1463536060053_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:40,263 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0, startTime=1463536060263, containerId=container_1463536060053_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:40,264 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536060053_0001, with dag id: dag_1463536060053_0001_1
2016-05-18 01:47:40,266 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060053_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40 Starting to run new task attempt: attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,267 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,267 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,267 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:40,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: BF07C4FF717249E9A9967A64882DA52E, VertexParallelism: 1, TaskAttemptID:attempt_1463536060053_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=2077A6B81DF44128B1AE45119C3B251F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E1CEA054BD9047869ADC1C7903BDC488, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:40,268 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:40,270 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,276 INFO input.MRInput (MRInput.java:initialize(440)) - 2077A6B81DF44128B1AE45119C3B251F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:40,277 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536060053_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:40,276 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:40,278 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536060053_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:40,277 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 2077A6B81DF44128B1AE45119C3B251F
2016-05-18 01:47:40,284 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BF07C4FF717249E9A9967A64882DA52E, ordinal: 0
2016-05-18 01:47:40,284 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BF07C4FF717249E9A9967A64882DA52E, mem on start (mb), free: 145, total: 490, max: 672
2016-05-18 01:47:40,284 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:40,284 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:40,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:40,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:40,285 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[2077A6B81DF44128B1AE45119C3B251F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E1CEA054BD9047869ADC1C7903BDC488:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:40,285 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:40,285 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:40,285 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:40,286 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[2077A6B81DF44128B1AE45119C3B251F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E1CEA054BD9047869ADC1C7903BDC488:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:40,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:40,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 2077A6B81DF44128B1AE45119C3B251F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:40,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:40,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:40,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 2077A6B81DF44128B1AE45119C3B251F
2016-05-18 01:47:40,287 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:40,287 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,290 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt:0+15, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt:15+15], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=30}
2016-05-18 01:47:40,292 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 2077A6B81DF44128B1AE45119C3B251F initialized RecordReader from event
2016-05-18 01:47:40,292 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 01:47:40,293 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"] streamed: true, id: 2077A6B81DF44128B1AE45119C3B251F
2016-05-18 01:47:40,293 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: F2B0D09410224E6CB09098CEE93D00DE
2016-05-18 01:47:40,294 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] F2B0D09410224E6CB09098CEE93D00DE
2016-05-18 01:47:40,295 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E1CEA054BD9047869ADC1C7903BDC488 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@61548d37, comparator=cascading.tuple.hadoop.util.TupleComparator@760347cc, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:40,296 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E1CEA054BD9047869ADC1C7903BDC488: , UsingHashComparator=false
2016-05-18 01:47:40,309 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:40,309 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:40,309 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E1CEA054BD9047869ADC1C7903BDC488: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:40,309 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E1CEA054BD9047869ADC1C7903BDC488: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:40,310 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lower+upper.txt"]
2016-05-18 01:47:40,310 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BF07C4FF717249E9A9967A64882DA52E, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:40,313 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,313 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BF07C4FF717249E9A9967A64882DA52E, mem on close (mb), free: 44, total: 490, max: 672
2016-05-18 01:47:40,313 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E1CEA054BD9047869ADC1C7903BDC488: Starting flush of map output
2016-05-18 01:47:40,313 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E1CEA054BD9047869ADC1C7903BDC488: Span0.length = 5, perItem = 21
2016-05-18 01:47:40,314 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E1CEA054BD9047869ADC1C7903BDC488: done sorting span=0, length=5, time=0
2016-05-18 01:47:40,314 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E1CEA054BD9047869ADC1C7903BDC488: Heap = SpanIterator<0:4> (span=Span[16777216,105]),
2016-05-18 01:47:40,316 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E1CEA054BD9047869ADC1C7903BDC488: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-local-dir/output/attempt_1463536060053_0001_1_00_000000_0_10037_0/file.out
2016-05-18 01:47:40,319 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:40 Completed running task attempt: attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,319 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:40,320 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536060053_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:40,320 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536060053_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536060311, Process_Duration=1, Process_End_Time=1463536060312, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-18 01:47:40,323 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536060053_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:40,323 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:40,323 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:40,323 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,323 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, taskAttemptId=attempt_1463536060053_0001_1_00_000000_0, creationTime=1463536060258, allocationTime=1463536060260, startTime=1463536060263, finishTime=1463536060323, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:40,324 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536060053_0001_1_00_000000_0
2016-05-18 01:47:40,323 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060053_0001_00_000001
2016-05-18 01:47:40,325 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_FINISHED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, taskId=task_1463536060053_0001_1_00_000000, startTime=1463536060263, finishTime=1463536060324, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1463536060053_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536060311, Process_Duration=1, Process_End_Time=1463536060312, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5
2016-05-18 01:47:40,326 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:40,326 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,327 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_FINISHED]: vertexName=BF07C4FF717249E9A9967A64882DA52E, vertexId=vertex_1463536060053_0001_1_00, initRequestedTime=1463536060231, initedTime=1463536060252, startRequestedTime=1463536060233, startedTime=1463536060253, finishTime=1463536060326, timeTaken=73, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536060311, Process_Duration=1, Process_End_Time=1463536060312, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463536060263, firstTasksToStart=[ task_1463536060053_0001_1_00_000000 ], lastTaskFinishTime=1463536060323, lastTasksToFinish=[ task_1463536060053_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536060053_0001_1_00_000000 ], longestDurationTasks=[ task_1463536060053_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:40,328 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:40,328 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536060053_0001_00_000001, stoppedTime=1463536060328, exitStatus=0
2016-05-18 01:47:40,328 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] attempt: attempt_1463536060053_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:40,329 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:40,329 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E1CEA054BD9047869ADC1C7903BDC488
2016-05-18 01:47:40,329 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:40,329 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BF07C4FF717249E9A9967A64882DA52E, destinationVertexName=E1CEA054BD9047869ADC1C7903BDC488. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:40,329 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,329 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536060053_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:40,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_STARTED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, taskId=task_1463536060053_0001_1_01_000000, scheduledTime=1463536060330, launchTime=1463536060330
2016-05-18 01:47:40,330 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,332 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536060053_0001_00_000002, launchTime=1463536060332
2016-05-18 01:47:40,333 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060053_0001_00_000002
2016-05-18 01:47:40,334 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060053_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,334 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536060053_0001_1_01_000000_0] started. Is using containerId: [container_1463536060053_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:40,334 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0, startTime=1463536060334, containerId=container_1463536060053_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:40 Starting to run new task attempt: attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,335 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,335 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,335 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:40,335 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536060053_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:40,336 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060053_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:40,335 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: E1CEA054BD9047869ADC1C7903BDC488, VertexParallelism: 1, TaskAttemptID:attempt_1463536060053_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BF07C4FF717249E9A9967A64882DA52E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6C371302A3E04EF8A0C3434252B2F1C4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:40,336 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536060053_0001_00_000001, Exiting
2016-05-18 01:47:40,336 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536060053_0001_00_000001
2016-05-18 01:47:40,336 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536060053_0001_00_000001
2016-05-18 01:47:40,336 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:40,337 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,338 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536060053_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:40,338 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536060053_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,338 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536060053_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:40,339 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536060053_0001_00_000001 completed successfully
2016-05-18 01:47:40,559 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:40,560 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:40,569 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E1CEA054BD9047869ADC1C7903BDC488, ordinal: 1
2016-05-18 01:47:40,569 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1CEA054BD9047869ADC1C7903BDC488, mem on start (mb), free: 342, total: 366, max: 672
2016-05-18 01:47:40,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:40,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:40,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:40,575 INFO output.MROutput (MROutput.java:initialize(421)) - 6C371302A3E04EF8A0C3434252B2F1C4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:40,575 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:40,575 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6C371302A3E04EF8A0C3434252B2F1C4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [BF07C4FF717249E9A9967A64882DA52E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:40,575 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:40,575 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:40,576 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:40,576 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6C371302A3E04EF8A0C3434252B2F1C4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [BF07C4FF717249E9A9967A64882DA52E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:40,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:40,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BF07C4FF717249E9A9967A64882DA52E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:40,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:40,576 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:40,577 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BF07C4FF717249E9A9967A64882DA52E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:40,577 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BF07C4FF717249E9A9967A64882DA52E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:40,579 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BF07C4FF717249E9A9967A64882DA52E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:40,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BF07C4FF717249E9A9967A64882DA52E
2016-05-18 01:47:40,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:40,580 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,582 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BF07C4FF717249E9A9967A64882DA52E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:40,583 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:40,584 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: F2B0D09410224E6CB09098CEE93D00DE
2016-05-18 01:47:40,585 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"], id: 6C371302A3E04EF8A0C3434252B2F1C4
2016-05-18 01:47:40,586 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536060053_0001_1_00_000000_0_10037} to DISK_DIRECT, csize=125, dsize=121, EndTime=1463536060585, TimeTaken=1, Rate=0.12 MB/s
2016-05-18 01:47:40,586 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BF07C4FF717249E9A9967A64882DA52E
2016-05-18 01:47:40,587 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:40,587 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-18 01:47:40,587 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] F2B0D09410224E6CB09098CEE93D00DE, for 1 inputs
2016-05-18 01:47:40,588 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BF07C4FF717249E9A9967A64882DA52E, wasInterrupted=false
2016-05-18 01:47:40,589 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:40,592 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:40,592 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001_wd/localmode-local-dir/output/attempt_1463536060053_0001_1_00_000000_0_10037/file.out, len=125, isLocal=true
2016-05-18 01:47:40,592 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 125 bytes from disk
2016-05-18 01:47:40,592 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:40,592 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:40,594 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2016-05-18 01:47:40,594 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E1CEA054BD9047869ADC1C7903BDC488, all 1 inputs ready in: 00:00:00.006
2016-05-18 01:47:40,594 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BF07C4FF717249E9A9967A64882DA52E
2016-05-18 01:47:40,594 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BF07C4FF717249E9A9967A64882DA52E: Shuffle Runner thread complete
2016-05-18 01:47:40,595 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BF07C4FF717249E9A9967A64882DA52E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:40,598 INFO output.MROutput (MROutput.java:close(526)) - 6C371302A3E04EF8A0C3434252B2F1C4 closed
2016-05-18 01:47:40,598 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536060053_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:40,599 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,599 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1CEA054BD9047869ADC1C7903BDC488, mem on close (mb), free: 339, total: 366, max: 672
2016-05-18 01:47:40,600 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BF07C4FF717249E9A9967A64882DA52E
2016-05-18 01:47:40,600 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BF07C4FF717249E9A9967A64882DA52E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:40,601 INFO output.MROutput (MROutput.java:close(526)) - 6C371302A3E04EF8A0C3434252B2F1C4 closed
2016-05-18 01:47:40,601 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:40 Completed running task attempt: attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,601 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:40,602 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536060053_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:40,602 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536060053_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536060594, Process_Duration=5, Process_End_Time=1463536060599, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 01:47:40,604 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536060053_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:40,604 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, taskAttemptId=attempt_1463536060053_0001_1_01_000000_0, creationTime=1463536060330, allocationTime=1463536060331, startTime=1463536060334, finishTime=1463536060604, timeTaken=270, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:40,604 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:40,604 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:40,604 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536060053_0001_1_01_000000_0
2016-05-18 01:47:40,605 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:TASK_FINISHED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, taskId=task_1463536060053_0001_1_01_000000, startTime=1463536060334, finishTime=1463536060604, timeTaken=270, status=SUCCEEDED, successfulAttemptID=attempt_1463536060053_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536060594, Process_Duration=5, Process_End_Time=1463536060599, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 01:47:40,605 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:40,605 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488]
2016-05-18 01:47:40,606 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:VERTEX_FINISHED]: vertexName=E1CEA054BD9047869ADC1C7903BDC488, vertexId=vertex_1463536060053_0001_1_01, initRequestedTime=1463536060233, initedTime=1463536060242, startRequestedTime=1463536060253, startedTime=1463536060253, finishTime=1463536060605, timeTaken=352, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536060594, Process_Duration=5, Process_End_Time=1463536060599, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463536060334, firstTasksToStart=[ task_1463536060053_0001_1_01_000000 ], lastTaskFinishTime=1463536060604, lastTasksToFinish=[ task_1463536060053_0001_1_01_000000 ], minTaskDuration=270, maxTaskDuration=270, avgTaskDuration=270.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536060053_0001_1_01_000000 ], longestDurationTasks=[ task_1463536060053_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:40,606 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:40,607 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536060053_0001_00_000002, stoppedTime=1463536060607, exitStatus=0
2016-05-18 01:47:40,607 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:40,607 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:40,607 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536060053_0001_1_00 [BF07C4FF717249E9A9967A64882DA52E]
2016-05-18 01:47:40,609 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,609 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060053_0001_00_000002
2016-05-18 01:47:40,609 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536060053_0001_1
2016-05-18 01:47:40,609 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536060053_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:40,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536060053_0001_1
2016-05-18 01:47:40,610 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060053_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:40,610 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060053_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:40,610 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536060053_0001_00_000002, Exiting
2016-05-18 01:47:40,610 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536060053_0001_00_000002
2016-05-18 01:47:40,610 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536060053_0001_00_000002
2016-05-18 01:47:40,610 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6C371302A3E04EF8A0C3434252B2F1C4 for vertex: vertex_1463536060053_0001_1_01 [E1CEA054BD9047869ADC1C7903BDC488], outputName: 6C371302A3E04EF8A0C3434252B2F1C4
2016-05-18 01:47:40,610 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536060053_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:40,611 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536060053_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:40,611 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536060053_0001_00_000002 completed successfully
2016-05-18 01:47:40,620 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6C371302A3E04EF8A0C3434252B2F1C4 of vertex/vertexGroup:E1CEA054BD9047869ADC1C7903BDC488 isVertexGroupOutput:false
2016-05-18 01:47:40,620 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:40,622 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536060053_0001_1, queueSize=0
2016-05-18 01:47:40,623 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060053_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536060053_0001_1, startTime=1463536060231, finishTime=1463536060620, timeTaken=389, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=780, AM_GC_TIME_MILLIS=217, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=10, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072120905, Process_Duration=6, Process_End_Time=2927072120911, Read_Duration=1, Tuples_Read=10, Tuples_Written=10, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=5, Tuples_Written=5
2016-05-18 01:47:40,623 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536060053_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:40,623 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060053_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:40 Completed Dag: dag_1463536060053_0001_1
2016-05-18 01:47:40,624 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536060053_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:40,624 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536060053_0001_1_post
2016-05-18 01:47:40,624 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:40,624 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:40,624 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:40,624 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3, with id=dag_1463536060053_0001_1
2016-05-18 01:47:40,625 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF] test/(1/1) ...testSimpleBuffer3/simple3, with id=dag_1463536060053_0001_1
2016-05-18 01:47:40,632 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463536060053_0001
2016-05-18 01:47:40,634 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:40,634 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:40,634 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:40,634 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:40,634 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:40,635 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:40,635 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:40,635 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:40,636 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:40,636 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:40,636 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:40,636 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:40,637 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:40,637 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:40,640 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/FEAF011A79AC4912BE22FAF164F4FE7C/FF546CA5F8F84D2085F6A7F8480AE1BF/.tez/application_1463536060053_0001
2016-05-18 01:47:40,641 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3/_temporary
2016-05-18 01:47:40,641 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.618
2016-05-18 01:47:40,723 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-18 01:47:40,753 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:40,754 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:40,817 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 01:47:40,830 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.032
2016-05-18 01:47:40,831 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:40,831 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:40,831 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:40,858 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:40,866 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-18 01:47:40,923 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-18 01:47:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:40,925 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferReverse/firstn
2016-05-18 01:47:40,925 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:40,942 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:40,946 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:40,947 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:40,947 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:40,956 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001 doesn't exist and is created
2016-05-18 01:47:40,975 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:40,976 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:40,977 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd
2016-05-18 01:47:40,992 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536060947_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:40,992 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:40,992 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:40,994 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:40,994 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:40,995 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:40,995 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:40,996 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:40,998 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:40,998 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:40,998 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:40,998 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:40,998 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:40,999 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:40,999 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:40,999 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:40,999 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:41,001 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:41,001 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:41,001 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:41,003 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:41,004 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:41,004 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:41,005 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-log-dir
2016-05-18 01:47:41,005 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-log-dir/history.txt.appattempt_1463536060947_0001_000000, maxErrors=10
2016-05-18 01:47:41,006 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:41,006 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536060947_0001_000000, appSubmitTime=1463536060991, launchTime=1463536060991
2016-05-18 01:47:41,010 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:41,013 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:46311
2016-05-18 01:47:41,015 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536060947
2016-05-18 01:47:41,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536060947_0001_000000, startTime=1463536061017
2016-05-18 01:47:41,017 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:41,056 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:41,056 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:41,077 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:41,077 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:41,077 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463536060947_0001, dagName=[A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 01:47:41,095 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 01:47:41,095 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:41,097 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536060947_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-log-dir/dag_1463536060947_0001_1.dot
2016-05-18 01:47:41,097 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-log-dir/dag_1463536060947_0001_1-tez-dag.pb.txt
2016-05-18 01:47:41,105 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536060947_0001_1
2016-05-18 01:47:41,105 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,105 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 01:47:41 Running Dag: dag_1463536060947_0001_1
2016-05-18 01:47:41,120 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536060947_0001_1, submitTime=1463536061095
2016-05-18 01:47:41,122 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001/tez-conf.pb" } size: 40937 timestamp: 1463536060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:41,122 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,122 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:41,123 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001/tez-conf.pb" } size: 40937 timestamp: 1463536060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:41,123 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,124 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:41,124 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:41,125 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536060947_0001_1, initTime=1463536061124
2016-05-18 01:47:41,125 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060947_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:41,125 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:41,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:DAG_STARTED]: dagID=dag_1463536060947_0001_1, startTime=1463536061126
2016-05-18 01:47:41,126 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060947_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:41,126 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463536060947_0001, dagName=[A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn
2016-05-18 01:47:41,126 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BD5E4B8C09904529B76B533E2221F1EA : {20543DE7BC70431F82F2CAF88438EFE2={InputName=20543DE7BC70431F82F2CAF88438EFE2}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:41,126 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 20543DE7BC70431F82F2CAF88438EFE2, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:41,127 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,127 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,127 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,127 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,128 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:41,128 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 20543DE7BC70431F82F2CAF88438EFE2 on vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,128 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,130 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 20543DE7BC70431F82F2CAF88438EFE2 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:41,133 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:41,133 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:41,133 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:41,133 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:41,134 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:41,134 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 20543DE7BC70431F82F2CAF88438EFE2 on vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,134 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,134 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536060947_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:41,136 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556], numAdditionalOutputs=1
2016-05-18 01:47:41,136 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=9264FCA2307E4C8E8A5B92F3514CC433, vertex=vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:41,139 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E589B2C1CB004708873B72EA8E72B556:9264FCA2307E4C8E8A5B92F3514CC433 using oldmapred API
2016-05-18 01:47:41,139 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=9264FCA2307E4C8E8A5B92F3514CC433, vertexName=E589B2C1CB004708873B72EA8E72B556, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:41,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E589B2C1CB004708873B72EA8E72B556, vertexId=vertex_1463536060947_0001_1_01, initRequestedTime=1463536061128, initedTime=1463536061136, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:41,141 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:41,142 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:41,142 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {20543DE7BC70431F82F2CAF88438EFE2=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:41,143 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] parallelism set to 1
2016-05-18 01:47:41,143 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:41,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, vertexId=vertex_1463536060947_0001_1_00, initRequestedTime=1463536061126, initedTime=1463536061144, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:41,144 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] with distanceFromRoot: 0
2016-05-18 01:47:41,144 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:41,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536060947_0001_1_00, startRequestedTime=1463536061128, startedTime=1463536061144
2016-05-18 01:47:41,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536060947_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:41,145 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:41,145 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536060947_0001_1_00 for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] numStartedSources: 1 numSources: 1
2016-05-18 01:47:41,145 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] with distanceFromRoot: 1
2016-05-18 01:47:41,145 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536060947_0001_1_01, startRequestedTime=1463536061145, startedTime=1463536061145
2016-05-18 01:47:41,145 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:41,149 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BD5E4B8C09904529B76B533E2221F1EA
2016-05-18 01:47:41,149 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E589B2C1CB004708873B72EA8E72B556 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:41,149 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_STARTED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, taskId=task_1463536060947_0001_1_00_000000, scheduledTime=1463536061150, launchTime=1463536061150
2016-05-18 01:47:41,151 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,151 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BD5E4B8C09904529B76B533E2221F1EA in vertex: E589B2C1CB004708873B72EA8E72B556 numBipartiteSourceTasks: 1
2016-05-18 01:47:41,154 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:41,156 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536060947_0001_00_000001, launchTime=1463536061156
2016-05-18 01:47:41,156 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060947_0001_00_000001
2016-05-18 01:47:41,157 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536060947_0001, with dag id: dag_1463536060947_0001_1
2016-05-18 01:47:41,157 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536060947_0001_1_00_000000_0] started. Is using containerId: [container_1463536060947_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:41,158 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0, startTime=1463536061157, containerId=container_1463536060947_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:41,159 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060947_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41 Starting to run new task attempt: attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,159 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,159 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,161 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:41,162 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn, VertexName: BD5E4B8C09904529B76B533E2221F1EA, VertexParallelism: 1, TaskAttemptID:attempt_1463536060947_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=20543DE7BC70431F82F2CAF88438EFE2, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E589B2C1CB004708873B72EA8E72B556, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:41,162 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:41,163 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,166 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536060947_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:41,166 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536060947_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:41,170 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:41,173 INFO input.MRInput (MRInput.java:initialize(440)) - 20543DE7BC70431F82F2CAF88438EFE2 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:41,173 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 20543DE7BC70431F82F2CAF88438EFE2
2016-05-18 01:47:41,185 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BD5E4B8C09904529B76B533E2221F1EA, ordinal: 0
2016-05-18 01:47:41,185 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BD5E4B8C09904529B76B533E2221F1EA, mem on start (mb), free: 232, total: 366, max: 672
2016-05-18 01:47:41,185 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:41,186 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:41,186 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:41,186 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:41,186 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[20543DE7BC70431F82F2CAF88438EFE2:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E589B2C1CB004708873B72EA8E72B556:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:41,188 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:41,188 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:41,188 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:41,188 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[20543DE7BC70431F82F2CAF88438EFE2:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E589B2C1CB004708873B72EA8E72B556:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:41,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:41,190 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 20543DE7BC70431F82F2CAF88438EFE2 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:41,190 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 20543DE7BC70431F82F2CAF88438EFE2
2016-05-18 01:47:41,190 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:41,190 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:41,190 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:41,191 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,193 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:41,195 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 20543DE7BC70431F82F2CAF88438EFE2 initialized RecordReader from event
2016-05-18 01:47:41,196 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:41,197 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 20543DE7BC70431F82F2CAF88438EFE2
2016-05-18 01:47:41,197 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 0B3C5AAE54304AFB9CE63D214D57911B
2016-05-18 01:47:41,197 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 0B3C5AAE54304AFB9CE63D214D57911B
2016-05-18 01:47:41,200 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E589B2C1CB004708873B72EA8E72B556 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.TuplePair, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3f0eca9f, comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@6418ad10, partitioner=cascading.tuple.tez.util.GroupingSortingPartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:41,200 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E589B2C1CB004708873B72EA8E72B556: , UsingHashComparator=false
2016-05-18 01:47:41,213 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:41,213 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:41,213 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E589B2C1CB004708873B72EA8E72B556: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:41,213 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E589B2C1CB004708873B72EA8E72B556: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:41,214 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:41,214 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BD5E4B8C09904529B76B533E2221F1EA, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:41,217 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,217 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BD5E4B8C09904529B76B533E2221F1EA, mem on close (mb), free: 130, total: 366, max: 672
2016-05-18 01:47:41,217 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E589B2C1CB004708873B72EA8E72B556: Starting flush of map output
2016-05-18 01:47:41,217 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E589B2C1CB004708873B72EA8E72B556: Span0.length = 13, perItem = 22
2016-05-18 01:47:41,218 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E589B2C1CB004708873B72EA8E72B556: done sorting span=0, length=13, time=1
2016-05-18 01:47:41,218 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E589B2C1CB004708873B72EA8E72B556: Heap = SpanIterator<0:12> (span=Span[16777216,286]),
2016-05-18 01:47:41,219 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E589B2C1CB004708873B72EA8E72B556: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-local-dir/output/attempt_1463536060947_0001_1_00_000000_0_10043_0/file.out
2016-05-18 01:47:41,223 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:41 Completed running task attempt: attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,224 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:41,224 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536060947_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:41,225 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536060947_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536061214, Process_Duration=3, Process_End_Time=1463536061217, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:41,225 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536060947_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,226 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,226 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:41,226 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, taskAttemptId=attempt_1463536060947_0001_1_00_000000_0, creationTime=1463536061150, allocationTime=1463536061155, startTime=1463536061157, finishTime=1463536061226, timeTaken=69, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:41,227 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536060947_0001_1_00_000000_0
2016-05-18 01:47:41,227 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060947_0001_00_000001
2016-05-18 01:47:41,227 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_FINISHED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, taskId=task_1463536060947_0001_1_00_000000, startTime=1463536061157, finishTime=1463536061227, timeTaken=70, status=SUCCEEDED, successfulAttemptID=attempt_1463536060947_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536061214, Process_Duration=3, Process_End_Time=1463536061217, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:41,228 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:41,228 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_FINISHED]: vertexName=BD5E4B8C09904529B76B533E2221F1EA, vertexId=vertex_1463536060947_0001_1_00, initRequestedTime=1463536061126, initedTime=1463536061144, startRequestedTime=1463536061128, startedTime=1463536061144, finishTime=1463536061228, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536061214, Process_Duration=3, Process_End_Time=1463536061217, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536061157, firstTasksToStart=[ task_1463536060947_0001_1_00_000000 ], lastTaskFinishTime=1463536061226, lastTasksToFinish=[ task_1463536060947_0001_1_00_000000 ], minTaskDuration=69, maxTaskDuration=69, avgTaskDuration=69.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536060947_0001_1_00_000000 ], longestDurationTasks=[ task_1463536060947_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:41,230 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:41,230 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536060947_0001_00_000001, stoppedTime=1463536061230, exitStatus=0
2016-05-18 01:47:41,230 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] attempt: attempt_1463536060947_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:41,230 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:41,230 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E589B2C1CB004708873B72EA8E72B556
2016-05-18 01:47:41,230 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:41,231 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BD5E4B8C09904529B76B533E2221F1EA, destinationVertexName=E589B2C1CB004708873B72EA8E72B556. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:41,231 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,231 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536060947_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:41,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_STARTED]: vertexName=E589B2C1CB004708873B72EA8E72B556, taskId=task_1463536060947_0001_1_01_000000, scheduledTime=1463536061231, launchTime=1463536061231
2016-05-18 01:47:41,232 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536060947_0001_00_000002, launchTime=1463536061233
2016-05-18 01:47:41,234 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060947_0001_00_000002
2016-05-18 01:47:41,236 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060947_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,237 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536060947_0001_1_01_000000_0] started. Is using containerId: [container_1463536060947_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:41,237 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E589B2C1CB004708873B72EA8E72B556, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0, startTime=1463536061236, containerId=container_1463536060947_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:41 Starting to run new task attempt: attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,237 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,237 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,238 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536060947_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:41,238 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060947_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:41,238 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536060947_0001_00_000001, Exiting
2016-05-18 01:47:41,238 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536060947_0001_00_000001
2016-05-18 01:47:41,238 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536060947_0001_00_000001
2016-05-18 01:47:41,239 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:41,239 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn, VertexName: E589B2C1CB004708873B72EA8E72B556, VertexParallelism: 1, TaskAttemptID:attempt_1463536060947_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BD5E4B8C09904529B76B533E2221F1EA, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=9264FCA2307E4C8E8A5B92F3514CC433, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:41,239 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:41,240 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,240 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536060947_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:41,241 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536060947_0001_00_000001 completed successfully
2016-05-18 01:47:41,242 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536060947_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,243 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536060947_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:41,251 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:41,254 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:41,262 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E589B2C1CB004708873B72EA8E72B556, ordinal: 1
2016-05-18 01:47:41,263 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E589B2C1CB004708873B72EA8E72B556, mem on start (mb), free: 335, total: 366, max: 672
2016-05-18 01:47:41,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:41,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:41,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:41,266 INFO output.MROutput (MROutput.java:initialize(421)) - 9264FCA2307E4C8E8A5B92F3514CC433: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:41,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:41,267 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[9264FCA2307E4C8E8A5B92F3514CC433:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [BD5E4B8C09904529B76B533E2221F1EA:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 01:47:41,267 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:41,267 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:41,267 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:41,268 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[9264FCA2307E4C8E8A5B92F3514CC433:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [BD5E4B8C09904529B76B533E2221F1EA:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 01:47:41,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:41,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BD5E4B8C09904529B76B533E2221F1EA being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:41,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:41,268 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:41,268 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BD5E4B8C09904529B76B533E2221F1EA: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:41,268 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BD5E4B8C09904529B76B533E2221F1EA: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:41,269 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BD5E4B8C09904529B76B533E2221F1EA with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:41,270 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BD5E4B8C09904529B76B533E2221F1EA
2016-05-18 01:47:41,270 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:41,271 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,271 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BD5E4B8C09904529B76B533E2221F1EA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:41,273 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536060947_0001_1_00_000000_0_10043} to DISK_DIRECT, csize=322, dsize=318, EndTime=1463536061273, TimeTaken=1, Rate=0.31 MB/s
2016-05-18 01:47:41,273 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BD5E4B8C09904529B76B533E2221F1EA
2016-05-18 01:47:41,273 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:41,274 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BD5E4B8C09904529B76B533E2221F1EA, wasInterrupted=false
2016-05-18 01:47:41,276 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:41,276 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:41,277 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:41,277 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 0B3C5AAE54304AFB9CE63D214D57911B
2016-05-18 01:47:41,277 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001_wd/localmode-local-dir/output/attempt_1463536060947_0001_1_00_000000_0_10043/file.out, len=322, isLocal=true
2016-05-18 01:47:41,278 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"], id: 9264FCA2307E4C8E8A5B92F3514CC433
2016-05-18 01:47:41,278 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 322 bytes from disk
2016-05-18 01:47:41,278 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:41,278 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-18 01:47:41,278 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:41,279 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 0B3C5AAE54304AFB9CE63D214D57911B, for 1 inputs
2016-05-18 01:47:41,279 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-18 01:47:41,279 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BD5E4B8C09904529B76B533E2221F1EA
2016-05-18 01:47:41,279 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BD5E4B8C09904529B76B533E2221F1EA: Shuffle Runner thread complete
2016-05-18 01:47:41,279 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E589B2C1CB004708873B72EA8E72B556, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:41,280 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BD5E4B8C09904529B76B533E2221F1EA: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator, keyClass=cascading.tuple.io.TuplePair, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:41,283 INFO output.MROutput (MROutput.java:close(526)) - 9264FCA2307E4C8E8A5B92F3514CC433 closed
2016-05-18 01:47:41,284 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536060947_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:41,284 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,284 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E589B2C1CB004708873B72EA8E72B556, mem on close (mb), free: 333, total: 366, max: 672
2016-05-18 01:47:41,285 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BD5E4B8C09904529B76B533E2221F1EA
2016-05-18 01:47:41,285 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BD5E4B8C09904529B76B533E2221F1EA: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:41,285 INFO output.MROutput (MROutput.java:close(526)) - 9264FCA2307E4C8E8A5B92F3514CC433 closed
2016-05-18 01:47:41,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:41 Completed running task attempt: attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,286 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:41,286 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536060947_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:41,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536060947_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536061279, Process_Duration=5, Process_End_Time=1463536061284, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 01:47:41,288 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536060947_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,288 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,288 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:41,288 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,288 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E589B2C1CB004708873B72EA8E72B556, taskAttemptId=attempt_1463536060947_0001_1_01_000000_0, creationTime=1463536061231, allocationTime=1463536061232, startTime=1463536061236, finishTime=1463536061288, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:41,288 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536060947_0001_00_000002
2016-05-18 01:47:41,289 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536060947_0001_1_01_000000_0
2016-05-18 01:47:41,289 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:TASK_FINISHED]: vertexName=E589B2C1CB004708873B72EA8E72B556, taskId=task_1463536060947_0001_1_01_000000, startTime=1463536061236, finishTime=1463536061289, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463536060947_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536061279, Process_Duration=5, Process_End_Time=1463536061284, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 01:47:41,290 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:41,290 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556]
2016-05-18 01:47:41,292 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:VERTEX_FINISHED]: vertexName=E589B2C1CB004708873B72EA8E72B556, vertexId=vertex_1463536060947_0001_1_01, initRequestedTime=1463536061128, initedTime=1463536061136, startRequestedTime=1463536061145, startedTime=1463536061145, finishTime=1463536061290, timeTaken=145, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536061279, Process_Duration=5, Process_End_Time=1463536061284, Read_Duration=2, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463536061236, firstTasksToStart=[ task_1463536060947_0001_1_01_000000 ], lastTaskFinishTime=1463536061288, lastTasksToFinish=[ task_1463536060947_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536060947_0001_1_01_000000 ], longestDurationTasks=[ task_1463536060947_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:41,292 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:41,292 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536060947_0001_00_000002, stoppedTime=1463536061292, exitStatus=0
2016-05-18 01:47:41,293 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:41,293 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:41,293 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536060947_0001_1_00 [BD5E4B8C09904529B76B533E2221F1EA]
2016-05-18 01:47:41,293 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536060947_0001_1
2016-05-18 01:47:41,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536060947_0001_1
2016-05-18 01:47:41,295 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 9264FCA2307E4C8E8A5B92F3514CC433 for vertex: vertex_1463536060947_0001_1_01 [E589B2C1CB004708873B72EA8E72B556], outputName: 9264FCA2307E4C8E8A5B92F3514CC433
2016-05-18 01:47:41,297 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060947_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:41,298 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536060947_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:41,299 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536060947_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:41,299 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536060947_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:41,300 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536060947_0001_00_000002, Exiting
2016-05-18 01:47:41,300 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536060947_0001_00_000002
2016-05-18 01:47:41,300 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536060947_0001_00_000002
2016-05-18 01:47:41,300 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536060947_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:41,300 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536060947_0001_00_000002 completed successfully
2016-05-18 01:47:41,305 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:9264FCA2307E4C8E8A5B92F3514CC433 of vertex/vertexGroup:E589B2C1CB004708873B72EA8E72B556 isVertexGroupOutput:false
2016-05-18 01:47:41,305 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:41,308 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536060947_0001_1, queueSize=0
2016-05-18 01:47:41,309 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536060947_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536060947_0001_1, startTime=1463536061125, finishTime=1463536061305, timeTaken=180, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=320, AM_GC_TIME_MILLIS=7, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=13, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072122493, Process_Duration=8, Process_End_Time=2927072122501, Read_Duration=3, Tuples_Read=18, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 01:47:41,310 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536060947_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:41,310 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536060947_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:41 Completed Dag: dag_1463536060947_0001_1
2016-05-18 01:47:41,312 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536060947_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:41,312 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536060947_0001_1_post
2016-05-18 01:47:41,313 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,313 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:41,313 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:41,314 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn, with id=dag_1463536060947_0001_1
2016-05-18 01:47:41,314 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70] test/(1/1) ...irstNBufferReverse/firstn, with id=dag_1463536060947_0001_1
2016-05-18 01:47:41,318 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463536060947_0001
2016-05-18 01:47:41,320 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:41,320 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:41,320 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:41,321 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:41,321 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:41,321 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:41,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:41,322 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:41,322 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:41,323 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:41,323 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:41,324 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:41,324 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:41,326 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:41,328 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/A06495971C9A4F9EA4FDD6A8C1C7CCC5/B2601D402CF5411EAC8C57AEF4137C70/.tez/application_1463536060947_0001
2016-05-18 01:47:41,330 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn/_temporary
2016-05-18 01:47:41,330 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.406
2016-05-18 01:47:41,410 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-18 01:47:41,445 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:41,445 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:41,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.013
2016-05-18 01:47:41,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.016
2016-05-18 01:47:41,490 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 01:47:41,490 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 01:47:41,521 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 01:47:41,522 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 01:47:41,573 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 01:47:41,597 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.038
2016-05-18 01:47:41,597 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:41,597 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 01:47:41,598 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 01:47:41,617 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt
2016-05-18 01:47:41,625 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn
2016-05-18 01:47:41,682 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-18 01:47:41,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:41,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-18 01:47:41,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-18 01:47:41,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-18 01:47:41,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-18 01:47:41,685 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testFirstNBuffer/firstn
2016-05-18 01:47:41,685 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 01:47:41,700 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-18 01:47:41,704 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:41,705 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 01:47:41,705 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 01:47:41,715 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001 doesn't exist and is created
2016-05-18 01:47:41,737 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 01:47:41,737 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 01:47:41,738 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd
2016-05-18 01:47:41,754 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463536061705_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 01:47:41,754 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:41,754 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.7.2, majorVersion=2, minorVersion=7
2016-05-18 01:47:41,756 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 01:47:41,756 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 01:47:41,756 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 01:47:41,756 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 01:47:41,757 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 01:47:41,759 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 01:47:41,759 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 01:47:41,759 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 01:47:41,760 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 01:47:41,760 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 01:47:41,760 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 01:47:41,760 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 01:47:41,761 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 01:47:41,761 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:41,761 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 01:47:41,761 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 01:47:41,762 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 01:47:41,764 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 01:47:41,764 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 01:47:41,765 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 01:47:41,765 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-log-dir
2016-05-18 01:47:41,766 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-log-dir/history.txt.appattempt_1463536061705_0001_000000, maxErrors=10
2016-05-18 01:47:41,766 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 01:47:41,766 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463536061705_0001_000000, appSubmitTime=1463536061753, launchTime=1463536061753
2016-05-18 01:47:41,775 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:53576
2016-05-18 01:47:41,775 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463536061705
2016-05-18 01:47:41,779 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 01:47:41,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463536061705_0001_000000, startTime=1463536061781
2016-05-18 01:47:41,781 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 01:47:41,837 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 01:47:41,837 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 01:47:41,837 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463536061705_0001, dagName=[4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 01:47:41,856 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 01:47:41,856 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 01:47:41,857 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463536061705_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-log-dir/dag_1463536061705_0001_1.dot
2016-05-18 01:47:41,858 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-log-dir/dag_1463536061705_0001_1-tez-dag.pb.txt
2016-05-18 01:47:41,872 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536061705_0001_1
2016-05-18 01:47:41,872 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,872 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 01:47:41 Running Dag: dag_1463536061705_0001_1
2016-05-18 01:47:41,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463536061705_0001_1, submitTime=1463536061856
2016-05-18 01:47:41,893 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001/tez-conf.pb" } size: 40937 timestamp: 1463536061000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:41,893 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,893 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:41,895 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001/tez-conf.pb" } size: 40937 timestamp: 1463536061000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5], Default Resources=<memory:1024, vCores:1>
2016-05-18 01:47:41,895 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:41,895 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 01:47:41,895 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 01:47:41,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463536061705_0001_1, initTime=1463536061895
2016-05-18 01:47:41,896 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536061705_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 01:47:41,897 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 01:47:41,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:DAG_STARTED]: dagID=dag_1463536061705_0001_1, startTime=1463536061897
2016-05-18 01:47:41,897 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536061705_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 01:47:41,898 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C5F43CD1055F45E18E5BCF9CB556590E : {760459D3FEC643AF8FCB85CF2548FE5D={InputName=760459D3FEC643AF8FCB85CF2548FE5D}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 01:47:41,898 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 760459D3FEC643AF8FCB85CF2548FE5D, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 01:47:41,898 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,898 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,898 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463536061705_0001, dagName=[4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-18 01:47:41,898 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,899 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,899 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 01:47:41,899 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 760459D3FEC643AF8FCB85CF2548FE5D on vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,899 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:41,902 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 760459D3FEC643AF8FCB85CF2548FE5D asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 01:47:41,904 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 01:47:41,904 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:41,904 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:41,905 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 01:47:41,905 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 52 Min splitLength: 52428800 New desired splits: 1 Total length: 52 Original splits: 2
2016-05-18 01:47:41,906 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 52 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 52 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 01:47:41,906 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 01:47:41,906 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 760459D3FEC643AF8FCB85CF2548FE5D on vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,906 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536061705_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 01:47:41,906 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5], numAdditionalOutputs=1
2016-05-18 01:47:41,906 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=86134D1505724C63A5D62289EE567DB7, vertex=vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 01:47:41,909 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B83DDD75E70146D49AA0C23081991AE5:86134D1505724C63A5D62289EE567DB7 using oldmapred API
2016-05-18 01:47:41,909 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=86134D1505724C63A5D62289EE567DB7, vertexName=B83DDD75E70146D49AA0C23081991AE5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 01:47:41,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, vertexId=vertex_1463536061705_0001_1_01, initRequestedTime=1463536061899, initedTime=1463536061906, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 01:47:41,910 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 01:47:41,911 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:41,911 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {760459D3FEC643AF8FCB85CF2548FE5D=forAllWorkUnits=true, update=[1]}
2016-05-18 01:47:41,912 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] parallelism set to 1
2016-05-18 01:47:41,912 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 01:47:41,913 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, vertexId=vertex_1463536061705_0001_1_00, initRequestedTime=1463536061898, initedTime=1463536061913, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 01:47:41,913 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] with distanceFromRoot: 0
2016-05-18 01:47:41,913 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 01:47:41,914 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536061705_0001_1_00, startRequestedTime=1463536061899, startedTime=1463536061914
2016-05-18 01:47:41,914 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463536061705_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 01:47:41,915 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:41,915 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463536061705_0001_1_00 for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] numStartedSources: 1 numSources: 1
2016-05-18 01:47:41,915 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] with distanceFromRoot: 1
2016-05-18 01:47:41,915 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C5F43CD1055F45E18E5BCF9CB556590E
2016-05-18 01:47:41,916 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,916 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463536061705_0001_1_01, startRequestedTime=1463536061915, startedTime=1463536061916
2016-05-18 01:47:41,916 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 01:47:41,917 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_STARTED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, taskId=task_1463536061705_0001_1_00_000000, scheduledTime=1463536061917, launchTime=1463536061917
2016-05-18 01:47:41,917 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:41,921 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 01:47:41,921 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B83DDD75E70146D49AA0C23081991AE5 with 0 source tasks and 1 pending tasks
2016-05-18 01:47:41,923 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C5F43CD1055F45E18E5BCF9CB556590E in vertex: B83DDD75E70146D49AA0C23081991AE5 numBipartiteSourceTasks: 1
2016-05-18 01:47:41,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536061705_0001_00_000001, launchTime=1463536061923
2016-05-18 01:47:41,923 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536061705_0001_00_000001
2016-05-18 01:47:41,925 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536061705_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,925 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536061705_0001_1_00_000000_0] started. Is using containerId: [container_1463536061705_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 01:47:41,925 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0, startTime=1463536061925, containerId=container_1463536061705_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 01:47:41 Starting to run new task attempt: attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,926 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,926 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,926 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:41,926 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: C5F43CD1055F45E18E5BCF9CB556590E, VertexParallelism: 1, TaskAttemptID:attempt_1463536061705_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=760459D3FEC643AF8FCB85CF2548FE5D, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B83DDD75E70146D49AA0C23081991AE5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 01:47:41,926 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:41,927 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,927 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463536061705_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 01:47:41,928 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536061705_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 01:47:41,930 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463536061705_0001, with dag id: dag_1463536061705_0001_1
2016-05-18 01:47:41,934 INFO input.MRInput (MRInput.java:initialize(440)) - 760459D3FEC643AF8FCB85CF2548FE5D using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 01:47:41,934 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 760459D3FEC643AF8FCB85CF2548FE5D
2016-05-18 01:47:41,936 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:41,944 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C5F43CD1055F45E18E5BCF9CB556590E, ordinal: 0
2016-05-18 01:47:41,944 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C5F43CD1055F45E18E5BCF9CB556590E, mem on start (mb), free: 206, total: 366, max: 672
2016-05-18 01:47:41,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:41,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:41,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:41,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:41,945 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[760459D3FEC643AF8FCB85CF2548FE5D:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B83DDD75E70146D49AA0C23081991AE5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 01:47:41,945 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:41,946 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:41,946 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 01:47:41,946 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[760459D3FEC643AF8FCB85CF2548FE5D:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B83DDD75E70146D49AA0C23081991AE5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 01:47:41,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:41,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 760459D3FEC643AF8FCB85CF2548FE5D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:41,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:41,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:41,947 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 760459D3FEC643AF8FCB85CF2548FE5D
2016-05-18 01:47:41,948 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:41,948 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,949 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:0+26, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt:26+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=52}
2016-05-18 01:47:41,950 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 760459D3FEC643AF8FCB85CF2548FE5D initialized RecordReader from event
2016-05-18 01:47:41,952 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:41,953 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"] streamed: true, id: 760459D3FEC643AF8FCB85CF2548FE5D
2016-05-18 01:47:41,953 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: D9E348825D0347AFB7347D51E2AC9422
2016-05-18 01:47:41,954 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] D9E348825D0347AFB7347D51E2AC9422
2016-05-18 01:47:41,955 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B83DDD75E70146D49AA0C23081991AE5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4c8878ba, comparator=cascading.tuple.hadoop.util.TupleComparator@13ae2297, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 01:47:41,956 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B83DDD75E70146D49AA0C23081991AE5: , UsingHashComparator=false
2016-05-18 01:47:41,980 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 01:47:41,981 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 01:47:41,981 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B83DDD75E70146D49AA0C23081991AE5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 01:47:41,981 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B83DDD75E70146D49AA0C23081991AE5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 01:47:41,981 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/classes/test/data/lhs.txt"]
2016-05-18 01:47:41,982 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C5F43CD1055F45E18E5BCF9CB556590E, all 1 inputs ready in: 00:00:00.000
2016-05-18 01:47:41,985 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,985 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C5F43CD1055F45E18E5BCF9CB556590E, mem on close (mb), free: 221, total: 355, max: 672
2016-05-18 01:47:41,986 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B83DDD75E70146D49AA0C23081991AE5: Starting flush of map output
2016-05-18 01:47:41,986 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B83DDD75E70146D49AA0C23081991AE5: Span0.length = 13, perItem = 15
2016-05-18 01:47:41,987 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B83DDD75E70146D49AA0C23081991AE5: done sorting span=0, length=13, time=1
2016-05-18 01:47:41,987 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B83DDD75E70146D49AA0C23081991AE5: Heap = SpanIterator<0:12> (span=Span[16777216,195]),
2016-05-18 01:47:41,989 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B83DDD75E70146D49AA0C23081991AE5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-local-dir/output/attempt_1463536061705_0001_1_00_000000_0_10049_0/file.out
2016-05-18 01:47:41,992 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:41 Completed running task attempt: attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,993 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0, askedToStop=false
2016-05-18 01:47:41,993 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536061705_0001_1_00_000000_0, stopRequested=false
2016-05-18 01:47:41,993 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536061705_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463536061982, Process_Duration=3, Process_End_Time=1463536061985, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-18 01:47:41,995 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536061705_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,995 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:41,995 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, taskAttemptId=attempt_1463536061705_0001_1_00_000000_0, creationTime=1463536061917, allocationTime=1463536061922, startTime=1463536061925, finishTime=1463536061995, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:41,996 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536061705_0001_1_00_000000_0
2016-05-18 01:47:41,995 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:41,996 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:41,997 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536061705_0001_00_000001
2016-05-18 01:47:41,998 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_FINISHED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, taskId=task_1463536061705_0001_1_00_000000, startTime=1463536061925, finishTime=1463536061996, timeTaken=71, status=SUCCEEDED, successfulAttemptID=attempt_1463536061705_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536061982, Process_Duration=3, Process_End_Time=1463536061985, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-18 01:47:41,998 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:41,998 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:42,000 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_FINISHED]: vertexName=C5F43CD1055F45E18E5BCF9CB556590E, vertexId=vertex_1463536061705_0001_1_00, initRequestedTime=1463536061898, initedTime=1463536061913, startRequestedTime=1463536061899, startedTime=1463536061914, finishTime=1463536061998, timeTaken=84, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463536061982, Process_Duration=3, Process_End_Time=1463536061985, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463536061925, firstTasksToStart=[ task_1463536061705_0001_1_00_000000 ], lastTaskFinishTime=1463536061995, lastTasksToFinish=[ task_1463536061705_0001_1_00_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536061705_0001_1_00_000000 ], longestDurationTasks=[ task_1463536061705_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:42,000 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:42,000 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536061705_0001_00_000001, stoppedTime=1463536062000, exitStatus=0
2016-05-18 01:47:42,001 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] attempt: attempt_1463536061705_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 01:47:42,001 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:42,001 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:42,001 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B83DDD75E70146D49AA0C23081991AE5
2016-05-18 01:47:42,001 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536061705_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 01:47:42,001 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C5F43CD1055F45E18E5BCF9CB556590E, destinationVertexName=B83DDD75E70146D49AA0C23081991AE5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 01:47:42,002 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:42,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_STARTED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, taskId=task_1463536061705_0001_1_01_000000, scheduledTime=1463536062002, launchTime=1463536062002
2016-05-18 01:47:42,002 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:42,004 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463536061705_0001_00_000002, launchTime=1463536062004
2016-05-18 01:47:42,004 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536061705_0001_00_000002
2016-05-18 01:47:42,005 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536061705_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,005 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463536061705_0001_1_01_000000_0] started. Is using containerId: [container_1463536061705_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 01:47:42,005 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0, startTime=1463536062005, containerId=container_1463536061705_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 01:47:42 Starting to run new task attempt: attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,006 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,006 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:42,007 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 01:47:42,007 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536061705_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 01:47:42,007 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: B83DDD75E70146D49AA0C23081991AE5, VertexParallelism: 1, TaskAttemptID:attempt_1463536061705_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C5F43CD1055F45E18E5BCF9CB556590E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=86134D1505724C63A5D62289EE567DB7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 01:47:42,007 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536061705_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:42,008 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536061705_0001_00_000001, Exiting
2016-05-18 01:47:42,008 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536061705_0001_00_000001
2016-05-18 01:47:42,008 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536061705_0001_00_000001
2016-05-18 01:47:42,009 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 01:47:42,009 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,009 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536061705_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:42,009 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536061705_0001_00_000001 completed successfully
2016-05-18 01:47:42,011 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 01:47:42,012 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463536061705_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:42,012 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463536061705_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 01:47:42,017 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 01:47:42,024 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B83DDD75E70146D49AA0C23081991AE5, ordinal: 1
2016-05-18 01:47:42,024 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B83DDD75E70146D49AA0C23081991AE5, mem on start (mb), free: 216, total: 355, max: 672
2016-05-18 01:47:42,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 01:47:42,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 01:47:42,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 01:47:42,027 INFO output.MROutput (MROutput.java:initialize(421)) - 86134D1505724C63A5D62289EE567DB7: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 01:47:42,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 01:47:42,028 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C5F43CD1055F45E18E5BCF9CB556590E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [86134D1505724C63A5D62289EE567DB7:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 01:47:42,028 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 01:47:42,028 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 01:47:42,029 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 01:47:42,030 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C5F43CD1055F45E18E5BCF9CB556590E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [86134D1505724C63A5D62289EE567DB7:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 01:47:42,030 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 01:47:42,030 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C5F43CD1055F45E18E5BCF9CB556590E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 01:47:42,031 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C5F43CD1055F45E18E5BCF9CB556590E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 01:47:42,032 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C5F43CD1055F45E18E5BCF9CB556590E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 01:47:42,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C5F43CD1055F45E18E5BCF9CB556590E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 01:47:42,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C5F43CD1055F45E18E5BCF9CB556590E
2016-05-18 01:47:42,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 01:47:42,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 01:47:42,033 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 01:47:42,034 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,035 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C5F43CD1055F45E18E5BCF9CB556590E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 01:47:42,036 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463536061705_0001_1_00_000000_0_10049} to DISK_DIRECT, csize=175, dsize=171, EndTime=1463536062036, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 01:47:42,037 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C5F43CD1055F45E18E5BCF9CB556590E
2016-05-18 01:47:42,037 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:42,037 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C5F43CD1055F45E18E5BCF9CB556590E, wasInterrupted=false
2016-05-18 01:47:42,037 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-18 01:47:42,038 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: D9E348825D0347AFB7347D51E2AC9422
2016-05-18 01:47:42,038 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"], id: 86134D1505724C63A5D62289EE567DB7
2016-05-18 01:47:42,038 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-18 01:47:42,039 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] D9E348825D0347AFB7347D51E2AC9422, for 1 inputs
2016-05-18 01:47:42,039 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 01:47:42,040 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 01:47:42,040 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001_wd/localmode-local-dir/output/attempt_1463536061705_0001_1_00_000000_0_10049/file.out, len=175, isLocal=true
2016-05-18 01:47:42,041 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 175 bytes from disk
2016-05-18 01:47:42,041 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 01:47:42,041 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 01:47:42,041 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-18 01:47:42,042 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C5F43CD1055F45E18E5BCF9CB556590E
2016-05-18 01:47:42,042 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B83DDD75E70146D49AA0C23081991AE5, all 1 inputs ready in: 00:00:00.003
2016-05-18 01:47:42,043 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C5F43CD1055F45E18E5BCF9CB556590E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 01:47:42,042 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C5F43CD1055F45E18E5BCF9CB556590E: Shuffle Runner thread complete
2016-05-18 01:47:42,046 INFO output.MROutput (MROutput.java:close(526)) - 86134D1505724C63A5D62289EE567DB7 closed
2016-05-18 01:47:42,046 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463536061705_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 01:47:42,047 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,047 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B83DDD75E70146D49AA0C23081991AE5, mem on close (mb), free: 213, total: 355, max: 672
2016-05-18 01:47:42,047 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C5F43CD1055F45E18E5BCF9CB556590E
2016-05-18 01:47:42,047 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C5F43CD1055F45E18E5BCF9CB556590E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 01:47:42,047 INFO output.MROutput (MROutput.java:close(526)) - 86134D1505724C63A5D62289EE567DB7 closed
2016-05-18 01:47:42,047 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 01:47:42 Completed running task attempt: attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,048 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0, askedToStop=false
2016-05-18 01:47:42,048 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463536061705_0001_1_01_000000_0, stopRequested=false
2016-05-18 01:47:42,049 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463536061705_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463536062042, Process_Duration=5, Process_End_Time=1463536062047, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-18 01:47:42,050 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463536061705_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:42,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, taskAttemptId=attempt_1463536061705_0001_1_01_000000_0, creationTime=1463536062002, allocationTime=1463536062003, startTime=1463536062005, finishTime=1463536062050, timeTaken=45, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 01:47:42,050 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 01:47:42,050 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463536061705_0001_1_01_000000_0
2016-05-18 01:47:42,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:TASK_FINISHED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, taskId=task_1463536061705_0001_1_01_000000, startTime=1463536062005, finishTime=1463536062050, timeTaken=45, status=SUCCEEDED, successfulAttemptID=attempt_1463536061705_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536062042, Process_Duration=5, Process_End_Time=1463536062047, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-18 01:47:42,051 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 01:47:42,051 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5]
2016-05-18 01:47:42,050 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 01:47:42,052 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:42,052 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463536061705_0001_00_000002
2016-05-18 01:47:42,053 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:VERTEX_FINISHED]: vertexName=B83DDD75E70146D49AA0C23081991AE5, vertexId=vertex_1463536061705_0001_1_01, initRequestedTime=1463536061899, initedTime=1463536061906, startRequestedTime=1463536061915, startedTime=1463536061916, finishTime=1463536062051, timeTaken=135, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463536062042, Process_Duration=5, Process_End_Time=1463536062047, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463536062005, firstTasksToStart=[ task_1463536061705_0001_1_01_000000 ], lastTaskFinishTime=1463536062050, lastTasksToFinish=[ task_1463536061705_0001_1_01_000000 ], minTaskDuration=45, maxTaskDuration=45, avgTaskDuration=45.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463536061705_0001_1_01_000000 ], longestDurationTasks=[ task_1463536061705_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 01:47:42,053 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 01:47:42,053 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463536061705_0001_00_000002, stoppedTime=1463536062053, exitStatus=0
2016-05-18 01:47:42,053 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 01:47:42,053 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:42,054 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463536061705_0001_1_00 [C5F43CD1055F45E18E5BCF9CB556590E]
2016-05-18 01:47:42,054 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463536061705_0001_1
2016-05-18 01:47:42,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463536061705_0001_1
2016-05-18 01:47:42,054 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 86134D1505724C63A5D62289EE567DB7 for vertex: vertex_1463536061705_0001_1_01 [B83DDD75E70146D49AA0C23081991AE5], outputName: 86134D1505724C63A5D62289EE567DB7
2016-05-18 01:47:42,054 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536061705_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 01:47:42,055 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463536061705_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 01:47:42,062 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463536061705_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 01:47:42,062 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463536061705_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 01:47:42,062 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463536061705_0001_00_000002, Exiting
2016-05-18 01:47:42,063 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463536061705_0001_00_000002
2016-05-18 01:47:42,063 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463536061705_0001_00_000002
2016-05-18 01:47:42,063 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463536061705_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 01:47:42,063 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463536061705_0001_00_000002 completed successfully
2016-05-18 01:47:42,064 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:86134D1505724C63A5D62289EE567DB7 of vertex/vertexGroup:B83DDD75E70146D49AA0C23081991AE5 isVertexGroupOutput:false
2016-05-18 01:47:42,064 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 01:47:42,066 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463536061705_0001_1, queueSize=0
2016-05-18 01:47:42,067 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463536061705_0001_1][Event:DAG_FINISHED]: dagId=dag_1463536061705_0001_1, startTime=1463536061897, finishTime=1463536062064, timeTaken=167, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=11, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=5, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927072124024, Process_Duration=8, Process_End_Time=2927072124032, Read_Duration=3, Tuples_Read=18, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-18 01:47:42,067 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463536061705_0001_1 finished with state: SUCCEEDED
2016-05-18 01:47:42,067 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463536061705_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 01:47:42 Completed Dag: dag_1463536061705_0001_1
2016-05-18 01:47:42,068 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463536061705_0001_1, dagState=SUCCEEDED
2016-05-18 01:47:42,068 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463536061705_0001_1_post
2016-05-18 01:47:42,068 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 01:47:42,068 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 01:47:42,068 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 01:47:42,068 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn, with id=dag_1463536061705_0001_1
2016-05-18 01:47:42,069 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2] test/(1/1) ...m/testFirstNBuffer/firstn, with id=dag_1463536061705_0001_1
2016-05-18 01:47:42,074 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463536061705_0001
2016-05-18 01:47:42,076 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 01:47:42,076 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 01:47:42,077 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 01:47:42,077 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 01:47:42,077 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 01:47:42,077 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 01:47:42,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 01:47:42,078 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 01:47:42,078 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 01:47:42,079 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 01:47:42,080 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 01:47:42,080 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 01:47:42,080 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 01:47:42,081 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 01:47:42,084 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/tmp/.staging/4F71343E9676492CB02EA6A103A8A5B5/AAC72A311F784D2184F47924ED732FD2/.tez/application_1463536061705_0001
2016-05-18 01:47:42,084 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn/_temporary
2016-05-18 01:47:42,085 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.402
2016-05-18 01:47:42,123 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 01:47:42,123 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 01:47:42,166 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.7.x-tez-0.8.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn